{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "967acdf8-b5f2-4746-a56f-28fdfc27595d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============GPU================\n",
      "Sat Dec 23 05:46:16 2023       \n",
      "+---------------------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |\n",
      "|-----------------------------------------+----------------------+----------------------+\n",
      "| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                                         |                      |               MIG M. |\n",
      "|=========================================+======================+======================|\n",
      "|   0  NVIDIA GeForce RTX 4080         On | 00000000:45:00.0 Off |                  N/A |\n",
      "|  0%   22C    P8               17W / 320W|      1MiB / 16376MiB |      0%      Default |\n",
      "|                                         |                      |                  N/A |\n",
      "+-----------------------------------------+----------------------+----------------------+\n",
      "                                                                                         \n",
      "+---------------------------------------------------------------------------------------+\n",
      "| Processes:                                                                            |\n",
      "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
      "|        ID   ID                                                             Usage      |\n",
      "|=======================================================================================|\n",
      "|  No running processes found                                                           |\n",
      "+---------------------------------------------------------------------------------------+\n",
      "============CUDA version================\n",
      "nvcc: NVIDIA (R) Cuda compiler driver\n",
      "Copyright (c) 2005-2022 NVIDIA Corporation\n",
      "Built on Wed_Sep_21_10:33:58_PDT_2022\n",
      "Cuda compilation tools, release 11.8, V11.8.89\n",
      "Build cuda_11.8.r11.8/compiler.31833905_0\n",
      "============CPU================\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "model name\t: AMD EPYC 7452 32-Core Processor\n",
      "============Memory================\n",
      "MemTotal:       263805516 kB\n"
     ]
    }
   ],
   "source": [
    "# GPU\n",
    "print(\"============GPU================\")\n",
    "!nvidia-smi\n",
    "\n",
    "# CUDA version\n",
    "print(\"============CUDA version================\")\n",
    "!nvcc --version\n",
    "\n",
    "# CPU\n",
    "print(\"============CPU================\")\n",
    "!cat /proc/cpuinfo | grep model\\ name\n",
    "\n",
    "# Memory\n",
    "print(\"============Memory================\")\n",
    "!cat /proc/meminfo | grep MemTotal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bb0c71b2-b7d2-47b2-82ab-24619929a13d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'llama'...\n",
      "remote: Enumerating objects: 417, done.\u001b[K\n",
      "remote: Counting objects: 100% (71/71), done.\u001b[K\n",
      "remote: Compressing objects: 100% (48/48), done.\u001b[K\n",
      "remote: Total 417 (delta 29), reused 49 (delta 15), pack-reused 346\u001b[K\n",
      "Receiving objects: 100% (417/417), 1.10 MiB | 11.59 MiB/s, done.\n",
      "Resolving deltas: 100% (214/214), done.\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/facebookresearch/llama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "080f46e7-5783-4fc1-9552-59f9021bdfc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/IPython/core/magics/osm.py:417: UserWarning: using dhist requires you to install the `pickleshare` library.\n",
      "  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]\n"
     ]
    }
   ],
   "source": [
    "%cd llama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e764773d-63c6-4076-bc0b-79583e7927b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading LICENSE and Acceptable Usage Policy\n",
      "--2023-12-23 05:46:31--  https://download.llamameta.net/LICENSE?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.63, 3.161.119.33, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable\n",
      "\n",
      "    The file is already fully retrieved; nothing to do.\n",
      "\n",
      "--2023-12-23 05:46:31--  https://download.llamameta.net/USE_POLICY.md?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.63, 3.161.119.33, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable\n",
      "\n",
      "    The file is already fully retrieved; nothing to do.\n",
      "\n",
      "Downloading tokenizer\n",
      "--2023-12-23 05:46:31--  https://download.llamameta.net/tokenizer.model?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.33, 3.161.119.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 499723 (488K) [binary/octet-stream]\n",
      "Saving to: ‘./tokenizer.model’\n",
      "\n",
      "./tokenizer.model   100%[===================>] 488.01K  --.-KB/s    in 0.04s   \n",
      "\n",
      "2023-12-23 05:46:31 (12.0 MB/s) - ‘./tokenizer.model’ saved [499723/499723]\n",
      "\n",
      "--2023-12-23 05:46:31--  https://download.llamameta.net/tokenizer_checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.23, 3.161.119.33, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 50 [binary/octet-stream]\n",
      "Saving to: ‘./tokenizer_checklist.chk’\n",
      "\n",
      "./tokenizer_checkli 100%[===================>]      50  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 05:46:31 (58.2 MB/s) - ‘./tokenizer_checklist.chk’ saved [50/50]\n",
      "\n",
      "tokenizer.model: OK\n",
      "Downloading llama-2-7b\n",
      "--2023-12-23 05:46:31--  https://download.llamameta.net/llama-2-7b/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.63, 3.161.119.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13476925163 (13G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-7b/consolidated.00.pth’\n",
      "\n",
      "./llama-2-7b/consol 100%[===================>]  12.55G  56.5MB/s    in 3m 50s  \n",
      "\n",
      "2023-12-23 05:50:22 (55.9 MB/s) - ‘./llama-2-7b/consolidated.00.pth’ saved [13476925163/13476925163]\n",
      "\n",
      "--2023-12-23 05:50:22--  https://download.llamameta.net/llama-2-7b/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.23, 3.161.119.33, 3.161.119.126, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.23|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 102 [application/json]\n",
      "Saving to: ‘./llama-2-7b/params.json’\n",
      "\n",
      "./llama-2-7b/params 100%[===================>]     102  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 05:50:23 (10.4 MB/s) - ‘./llama-2-7b/params.json’ saved [102/102]\n",
      "\n",
      "--2023-12-23 05:50:23--  https://download.llamameta.net/llama-2-7b/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.33, 3.161.119.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 100 [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-7b/checklist.chk’\n",
      "\n",
      "./llama-2-7b/checkl 100%[===================>]     100  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 05:50:25 (6.48 MB/s) - ‘./llama-2-7b/checklist.chk’ saved [100/100]\n",
      "\n",
      "Checking checksums\n",
      "consolidated.00.pth: OK\n",
      "params.json: OK\n",
      "Downloading llama-2-13b\n",
      "--2023-12-23 05:50:48--  https://download.llamameta.net/llama-2-13b/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.23, 3.161.119.33, 3.161.119.126, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.23|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13016329643 (12G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-13b/consolidated.00.pth’\n",
      "\n",
      "./llama-2-13b/conso 100%[===================>]  12.12G  48.3MB/s    in 3m 45s  \n",
      "\n",
      "2023-12-23 05:54:33 (55.2 MB/s) - ‘./llama-2-13b/consolidated.00.pth’ saved [13016329643/13016329643]\n",
      "\n",
      "--2023-12-23 05:54:33--  https://download.llamameta.net/llama-2-13b/consolidated.01.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.63, 3.161.119.23, 3.161.119.126, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.63|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 13016329643 (12G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-13b/consolidated.01.pth’\n",
      "\n",
      "./llama-2-13b/conso 100%[===================>]  12.12G  38.5MB/s    in 3m 31s  \n",
      "\n",
      "2023-12-23 05:58:05 (58.9 MB/s) - ‘./llama-2-13b/consolidated.01.pth’ saved [13016329643/13016329643]\n",
      "\n",
      "--2023-12-23 05:58:05--  https://download.llamameta.net/llama-2-13b/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.63, 3.161.119.33, 3.161.119.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.63|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 102 [application/json]\n",
      "Saving to: ‘./llama-2-13b/params.json’\n",
      "\n",
      "./llama-2-13b/param 100%[===================>]     102  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 05:58:05 (10.5 MB/s) - ‘./llama-2-13b/params.json’ saved [102/102]\n",
      "\n",
      "--2023-12-23 05:58:05--  https://download.llamameta.net/llama-2-13b/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.33, 3.161.119.63, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 154 [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-13b/checklist.chk’\n",
      "\n",
      "./llama-2-13b/check 100%[===================>]     154  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 05:58:06 (13.3 MB/s) - ‘./llama-2-13b/checklist.chk’ saved [154/154]\n",
      "\n",
      "Checking checksums\n",
      "consolidated.00.pth: OK\n",
      "consolidated.01.pth: OK\n",
      "params.json: OK\n",
      "Downloading llama-2-70b\n",
      "--2023-12-23 05:58:50--  https://download.llamameta.net/llama-2-70b/consolidated.00.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.33, 3.161.119.23, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.00.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  65.9MB/s    in 5m 42s  \n",
      "\n",
      "2023-12-23 06:04:32 (48.2 MB/s) - ‘./llama-2-70b/consolidated.00.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:04:32--  https://download.llamameta.net/llama-2-70b/consolidated.01.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.33, 3.161.119.63, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.01.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  55.0MB/s    in 4m 57s  \n",
      "\n",
      "2023-12-23 06:09:29 (55.4 MB/s) - ‘./llama-2-70b/consolidated.01.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:09:29--  https://download.llamameta.net/llama-2-70b/consolidated.02.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.126, 3.161.119.63, 3.161.119.33, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.126|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.02.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  64.1MB/s    in 4m 28s  \n",
      "\n",
      "2023-12-23 06:13:58 (61.3 MB/s) - ‘./llama-2-70b/consolidated.02.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:13:58--  https://download.llamameta.net/llama-2-70b/consolidated.03.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.33, 3.161.119.63, 3.161.119.126, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.33|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.03.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  69.5MB/s    in 4m 3s   \n",
      "\n",
      "2023-12-23 06:18:01 (67.8 MB/s) - ‘./llama-2-70b/consolidated.03.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:18:01--  https://download.llamameta.net/llama-2-70b/consolidated.04.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 13.32.110.127, 13.32.110.49, 13.32.110.124, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|13.32.110.127|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.04.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  64.9MB/s    in 3m 58s  \n",
      "\n",
      "2023-12-23 06:22:00 (69.0 MB/s) - ‘./llama-2-70b/consolidated.04.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:22:00--  https://download.llamameta.net/llama-2-70b/consolidated.05.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 18.66.27.56, 18.66.27.85, 18.66.27.65, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|18.66.27.56|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.05.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  70.0MB/s    in 4m 2s   \n",
      "\n",
      "2023-12-23 06:26:02 (68.0 MB/s) - ‘./llama-2-70b/consolidated.05.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:26:02--  https://download.llamameta.net/llama-2-70b/consolidated.06.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 3.161.119.23, 3.161.119.126, 3.161.119.33, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|3.161.119.23|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.06.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  68.1MB/s    in 4m 6s   \n",
      "\n",
      "2023-12-23 06:30:09 (66.9 MB/s) - ‘./llama-2-70b/consolidated.06.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:30:09--  https://download.llamameta.net/llama-2-70b/consolidated.07.pth?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 18.66.27.56, 18.66.27.10, 18.66.27.85, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|18.66.27.56|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 17246706245 (16G) [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/consolidated.07.pth’\n",
      "\n",
      "./llama-2-70b/conso 100%[===================>]  16.06G  55.4MB/s    in 4m 22s  \n",
      "\n",
      "2023-12-23 06:34:32 (62.7 MB/s) - ‘./llama-2-70b/consolidated.07.pth’ saved [17246706245/17246706245]\n",
      "\n",
      "--2023-12-23 06:34:32--  https://download.llamameta.net/llama-2-70b/params.json?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 18.66.27.10, 18.66.27.56, 18.66.27.85, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|18.66.27.10|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 147 [application/json]\n",
      "Saving to: ‘./llama-2-70b/params.json’\n",
      "\n",
      "./llama-2-70b/param 100%[===================>]     147  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 06:34:33 (175 MB/s) - ‘./llama-2-70b/params.json’ saved [147/147]\n",
      "\n",
      "--2023-12-23 06:34:33--  https://download.llamameta.net/llama-2-70b/checklist.chk?Policy=eyJTdGF0ZW1lbnQiOlt7InVuaXF1ZV9oYXNoIjoiaGltMW5wdWVnZXYyM2Zpbml3c3Q1eXRxIiwiUmVzb3VyY2UiOiJodHRwczpcL1wvZG93bmxvYWQubGxhbWFtZXRhLm5ldFwvKiIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcwMzM5MTE1N319fV19&Signature=OVV5eCpY35fBCM7qlNcUdWqo4bUKbhg3p04wnwIbxq27GBpwFZuJSpX8lLzsO4SHNjG4DTz0bIQmG5IqUalnYqp56bao2fKPMAlnIuotvH-fWKAbg1BfW8hC9nh6SoZFIjMJNoW96sL2oTMbHS-Djxn-4xYZrdmfK5a4bTEL127ZaXbbVzftAeUku7p7Qe20QlOVDQjLbogXU0ElK-EVH3n-jjjU2Re154y2aLITcZnitNFa8dcQHHvPSozKi2erSQ%7E5f0crgyIF4upOSVYi6uiZgn5a2MqhY-TiXWS94mH0bx4n2ITENl7atSwBY36tEq5Wfr47H4YcvH7PCipS9A__&Key-Pair-Id=K15QRJLYKIFSLZ&Download-Request-ID=819074826636455\n",
      "Resolving download.llamameta.net (download.llamameta.net)... 18.66.27.10, 18.66.27.56, 18.66.27.85, ...\n",
      "Connecting to download.llamameta.net (download.llamameta.net)|18.66.27.10|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 478 [binary/octet-stream]\n",
      "Saving to: ‘./llama-2-70b/checklist.chk’\n",
      "\n",
      "./llama-2-70b/check 100%[===================>]     478  --.-KB/s    in 0s      \n",
      "\n",
      "2023-12-23 06:34:34 (38.6 MB/s) - ‘./llama-2-70b/checklist.chk’ saved [478/478]\n",
      "\n",
      "Checking checksums\n",
      "consolidated.00.pth: OK\n",
      "consolidated.01.pth: OK\n",
      "consolidated.02.pth: OK\n",
      "consolidated.03.pth: OK\n",
      "consolidated.04.pth: OK\n",
      "consolidated.05.pth: OK\n",
      "consolidated.06.pth: OK\n",
      "consolidated.07.pth: OK\n",
      "params.json: OK\n"
     ]
    }
   ],
   "source": [
    "# Define your PRESIGNED_URL and MODEL_SIZE in the script to prevent asking in the notebook\n",
    "!bash download.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1541dcc9-b822-4783-b6eb-020fc4a0316d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace\n"
     ]
    }
   ],
   "source": [
    "%cd /workspace\n",
    "!mkdir -p llama.cpp/models/7B-v2/\n",
    "!mv llama/llama-2-7b/* llama.cpp/models/7B-v2/\n",
    "!mkdir -p llama.cpp/models/13B-v2/\n",
    "!mv llama/llama-2-13b/* llama.cpp/models/13B-v2/\n",
    "!mkdir -p llama.cpp/models/70B-v2/\n",
    "!mv llama/llama-2-70b/* llama.cpp/models/70B-v2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cea2bab8-7c0d-42cc-8e32-064e71a58a74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama.cpp\n"
     ]
    }
   ],
   "source": [
    "%cd llama.cpp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9594719a-ea2a-4b8d-bd26-2c19f0a6a2de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 13283  100 13283    0     0  47318      0 --:--:-- --:--:-- --:--:-- 47439\n"
     ]
    }
   ],
   "source": [
    "# If you encounter the error \"does not appear to have a file named config.json\" when converting the models to ggml FP16 format, try to convert the model to huggingface format to get the config.json file.\n",
    "!curl -o convert_llama_weights_to_hf.py https://raw.githubusercontent.com/huggingface/transformers/main/src/transformers/models/llama/convert_llama_weights_to_hf.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "06b44ccf-4bf6-47a4-8f05-87a662822110",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama.cpp/models\n"
     ]
    }
   ],
   "source": [
    "%cd models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8f541028-baaa-40f8-8e1c-c359b5ead34c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp tokenizer.model 7B-v2/\n",
    "!cp tokenizer.model 13B-v2/\n",
    "!cp tokenizer.model 70B-v2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fcc944e7-32a8-4918-8001-6b51fb835377",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspace/llama.cpp\n"
     ]
    }
   ],
   "source": [
    "%cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ae14bc36-f1ba-4069-82bc-63242471a393",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting numpy==1.24.4 (from -r requirements.txt (line 1))\n",
      "  Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)\n",
      "Collecting sentencepiece==0.1.98 (from -r requirements.txt (line 2))\n",
      "  Downloading sentencepiece-0.1.98-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting transformers>=4.34.0 (from -r requirements.txt (line 3))\n",
      "  Downloading transformers-4.36.2-py3-none-any.whl.metadata (126 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting gguf>=0.1.0 (from -r requirements.txt (line 4))\n",
      "  Downloading gguf-0.6.0-py3-none-any.whl.metadata (3.2 kB)\n",
      "Collecting protobuf>=4.21.0 (from -r requirements.txt (line 5))\n",
      "  Downloading protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (3.9.0)\n",
      "Collecting huggingface-hub<1.0,>=0.19.3 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Downloading huggingface_hub-0.20.1-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (6.0.1)\n",
      "Collecting regex!=2019.12.17 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Downloading regex-2023.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34.0->-r requirements.txt (line 3)) (2.31.0)\n",
      "Collecting tokenizers<0.19,>=0.14 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Downloading tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
      "Collecting safetensors>=0.3.1 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Downloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
      "Collecting tqdm>=4.27 (from transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Downloading tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.6/57.6 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.19.3->transformers>=4.34.0->-r requirements.txt (line 3))\n",
      "  Downloading fsspec-2023.12.2-py3-none-any.whl.metadata (6.8 kB)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.19.3->transformers>=4.34.0->-r requirements.txt (line 3)) (4.4.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (2.1.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (1.26.13)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.34.0->-r requirements.txt (line 3)) (2022.12.7)\n",
      "Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.3/17.3 MB\u001b[0m \u001b[31m48.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading transformers-4.36.2-py3-none-any.whl (8.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.2/8.2 MB\u001b[0m \u001b[31m48.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading gguf-0.6.0-py3-none-any.whl (23 kB)\n",
      "Downloading protobuf-4.25.1-cp37-abi3-manylinux2014_x86_64.whl (294 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.6/294.6 kB\u001b[0m \u001b[31m62.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading huggingface_hub-0.20.1-py3-none-any.whl (330 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m64.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading regex-2023.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (773 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m773.9/773.9 kB\u001b[0m \u001b[31m45.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading safetensors-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m36.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.8/3.8 MB\u001b[0m \u001b[31m20.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m24.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading fsspec-2023.12.2-py3-none-any.whl (168 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m169.0/169.0 kB\u001b[0m \u001b[31m48.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: sentencepiece, tqdm, safetensors, regex, protobuf, numpy, fsspec, huggingface-hub, gguf, tokenizers, transformers\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 1.24.1\n",
      "    Uninstalling numpy-1.24.1:\n",
      "      Successfully uninstalled numpy-1.24.1\n",
      "  Attempting uninstall: fsspec\n",
      "    Found existing installation: fsspec 2023.4.0\n",
      "    Uninstalling fsspec-2023.4.0:\n",
      "      Successfully uninstalled fsspec-2023.4.0\n",
      "Successfully installed fsspec-2023.12.2 gguf-0.6.0 huggingface-hub-0.20.1 numpy-1.24.4 protobuf-4.25.1 regex-2023.10.3 safetensors-0.4.1 sentencepiece-0.1.98 tokenizers-0.15.0 tqdm-4.66.1 transformers-4.36.2\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3 -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# !pip uninstall accelerate # If you have this package, uninstall it first, then use `convert to hf model` to get the config.json.\n",
    "# install Python dependencies\n",
    "!python3 -m pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a95b3a38-17fb-4792-b1cd-4255e6ed2a7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Fetching all parameters from the checkpoint at models/7B-v2/.\n",
      "Loading the checkpoint in a Llama model.\n",
      "Traceback (most recent call last):\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 319, in <module>\n",
      "    main()\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 307, in main\n",
      "    write_model(\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 271, in write_model\n",
      "    model = LlamaForCausalLM.from_pretrained(tmp_model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True)\n",
      "  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 2863, in from_pretrained\n",
      "    raise ImportError(\n",
      "ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Fetching all parameters from the checkpoint at models/13B-v2/.\n",
      "Loading the checkpoint in a Llama model.\n",
      "Traceback (most recent call last):\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 319, in <module>\n",
      "    main()\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 307, in main\n",
      "    write_model(\n",
      "  File \"/workspace/llama.cpp/convert_llama_weights_to_hf.py\", line 271, in write_model\n",
      "    model = LlamaForCausalLM.from_pretrained(tmp_model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True)\n",
      "  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 2863, in from_pretrained\n",
      "    raise ImportError(\n",
      "ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Fetching all parameters from the checkpoint at models/70B-v2/.\n"
     ]
    }
   ],
   "source": [
    "# We don't need these models actually. We only need this to figure out the config.json error.\n",
    "!python3 convert_llama_weights_to_hf.py --input_dir models/7B-v2/ --model_size 7B --output_dir models/7B-v2/\n",
    "!python3 convert_llama_weights_to_hf.py --input_dir models/13B-v2/ --model_size 13B --output_dir models/13B-v2/\n",
    "!python3 convert_llama_weights_to_hf.py --input_dir models/70B-v2/ --model_size 70B --output_dir models/70B-v2/ # Surprisingly, it still solves the problem although you can't find the config.json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "839375fa-44f5-498c-8f1e-0e22ad8311ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit your params.json file if the \"vocab_size\" mismatch\n",
    "import json\n",
    "\n",
    "# Load the JSON file\n",
    "with open('models/7B-v2/params.json', 'r') as file:\n",
    "    data = json.load(file)\n",
    "\n",
    "# Modify the 'vocab_size' key\n",
    "data['vocab_size'] = 32000\n",
    "\n",
    "# Write the modified data back to the file\n",
    "with open('models/7B-v2/params.json', 'w') as file:\n",
    "    json.dump(data, file, indent=4)\n",
    "\n",
    "# Load the JSON file\n",
    "with open('models/13B-v2/params.json', 'r') as file:\n",
    "    data = json.load(file)\n",
    "\n",
    "# Modify the 'vocab_size' key\n",
    "data['vocab_size'] = 32000\n",
    "\n",
    "# Write the modified data back to the file\n",
    "with open('models/13B-v2/params.json', 'w') as file:\n",
    "    json.dump(data, file, indent=4)\n",
    "\n",
    "# Load the JSON file\n",
    "with open('models/70B-v2/params.json', 'r') as file:\n",
    "    data = json.load(file)\n",
    "\n",
    "# Modify the 'vocab_size' key\n",
    "data['vocab_size'] = 32000\n",
    "\n",
    "# Write the modified data back to the file\n",
    "with open('models/70B-v2/params.json', 'w') as file:\n",
    "    json.dump(data, file, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c5ce9c63-6f03-4736-a1df-56b9605f698b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading model file models/7B-v2/consolidated.00.pth\n",
      "params = Params(n_vocab=32000, n_embd=4096, n_layer=32, n_ctx=4096, n_ff=11008, n_head=32, n_head_kv=32, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('models/7B-v2'))\n",
      "32000 32000\n",
      "Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>\n",
      "Special vocab info: <SpecialVocab with 61249 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0}, add special tokens {'bos': True, 'eos': False}>\n",
      "tok_embeddings.weight                            -> token_embd.weight                        | BF16   | [32000, 4096]\n",
      "norm.weight                                      -> output_norm.weight                       | BF16   | [4096]\n",
      "output.weight                                    -> output.weight                            | BF16   | [32000, 4096]\n",
      "layers.0.attention.wq.weight                     -> blk.0.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.0.attention.wk.weight                     -> blk.0.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.0.attention.wv.weight                     -> blk.0.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.0.attention.wo.weight                     -> blk.0.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.0.feed_forward.w1.weight                  -> blk.0.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.0.feed_forward.w2.weight                  -> blk.0.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.0.feed_forward.w3.weight                  -> blk.0.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.0.attention_norm.weight                   -> blk.0.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.0.ffn_norm.weight                         -> blk.0.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.1.attention.wq.weight                     -> blk.1.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.1.attention.wk.weight                     -> blk.1.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.1.attention.wv.weight                     -> blk.1.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.1.attention.wo.weight                     -> blk.1.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.1.feed_forward.w1.weight                  -> blk.1.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.1.feed_forward.w2.weight                  -> blk.1.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.1.feed_forward.w3.weight                  -> blk.1.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.1.attention_norm.weight                   -> blk.1.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.1.ffn_norm.weight                         -> blk.1.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.2.attention.wq.weight                     -> blk.2.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.2.attention.wk.weight                     -> blk.2.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.2.attention.wv.weight                     -> blk.2.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.2.attention.wo.weight                     -> blk.2.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.2.feed_forward.w1.weight                  -> blk.2.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.2.feed_forward.w2.weight                  -> blk.2.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.2.feed_forward.w3.weight                  -> blk.2.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.2.attention_norm.weight                   -> blk.2.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.2.ffn_norm.weight                         -> blk.2.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.3.attention.wq.weight                     -> blk.3.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.3.attention.wk.weight                     -> blk.3.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.3.attention.wv.weight                     -> blk.3.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.3.attention.wo.weight                     -> blk.3.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.3.feed_forward.w1.weight                  -> blk.3.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.3.feed_forward.w2.weight                  -> blk.3.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.3.feed_forward.w3.weight                  -> blk.3.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.3.attention_norm.weight                   -> blk.3.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.3.ffn_norm.weight                         -> blk.3.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.4.attention.wq.weight                     -> blk.4.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.4.attention.wk.weight                     -> blk.4.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.4.attention.wv.weight                     -> blk.4.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.4.attention.wo.weight                     -> blk.4.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.4.feed_forward.w1.weight                  -> blk.4.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.4.feed_forward.w2.weight                  -> blk.4.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.4.feed_forward.w3.weight                  -> blk.4.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.4.attention_norm.weight                   -> blk.4.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.4.ffn_norm.weight                         -> blk.4.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.5.attention.wq.weight                     -> blk.5.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.5.attention.wk.weight                     -> blk.5.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.5.attention.wv.weight                     -> blk.5.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.5.attention.wo.weight                     -> blk.5.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.5.feed_forward.w1.weight                  -> blk.5.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.5.feed_forward.w2.weight                  -> blk.5.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.5.feed_forward.w3.weight                  -> blk.5.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.5.attention_norm.weight                   -> blk.5.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.5.ffn_norm.weight                         -> blk.5.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.6.attention.wq.weight                     -> blk.6.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.6.attention.wk.weight                     -> blk.6.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.6.attention.wv.weight                     -> blk.6.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.6.attention.wo.weight                     -> blk.6.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.6.feed_forward.w1.weight                  -> blk.6.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.6.feed_forward.w2.weight                  -> blk.6.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.6.feed_forward.w3.weight                  -> blk.6.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.6.attention_norm.weight                   -> blk.6.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.6.ffn_norm.weight                         -> blk.6.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.7.attention.wq.weight                     -> blk.7.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.7.attention.wk.weight                     -> blk.7.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.7.attention.wv.weight                     -> blk.7.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.7.attention.wo.weight                     -> blk.7.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.7.feed_forward.w1.weight                  -> blk.7.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.7.feed_forward.w2.weight                  -> blk.7.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.7.feed_forward.w3.weight                  -> blk.7.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.7.attention_norm.weight                   -> blk.7.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.7.ffn_norm.weight                         -> blk.7.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.8.attention.wq.weight                     -> blk.8.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.8.attention.wk.weight                     -> blk.8.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.8.attention.wv.weight                     -> blk.8.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.8.attention.wo.weight                     -> blk.8.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.8.feed_forward.w1.weight                  -> blk.8.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.8.feed_forward.w2.weight                  -> blk.8.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.8.feed_forward.w3.weight                  -> blk.8.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.8.attention_norm.weight                   -> blk.8.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.8.ffn_norm.weight                         -> blk.8.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.9.attention.wq.weight                     -> blk.9.attn_q.weight                      | BF16   | [4096, 4096]\n",
      "layers.9.attention.wk.weight                     -> blk.9.attn_k.weight                      | BF16   | [4096, 4096]\n",
      "layers.9.attention.wv.weight                     -> blk.9.attn_v.weight                      | BF16   | [4096, 4096]\n",
      "layers.9.attention.wo.weight                     -> blk.9.attn_output.weight                 | BF16   | [4096, 4096]\n",
      "layers.9.feed_forward.w1.weight                  -> blk.9.ffn_gate.weight                    | BF16   | [11008, 4096]\n",
      "layers.9.feed_forward.w2.weight                  -> blk.9.ffn_down.weight                    | BF16   | [4096, 11008]\n",
      "layers.9.feed_forward.w3.weight                  -> blk.9.ffn_up.weight                      | BF16   | [11008, 4096]\n",
      "layers.9.attention_norm.weight                   -> blk.9.attn_norm.weight                   | BF16   | [4096]\n",
      "layers.9.ffn_norm.weight                         -> blk.9.ffn_norm.weight                    | BF16   | [4096]\n",
      "layers.10.attention.wq.weight                    -> blk.10.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.10.attention.wk.weight                    -> blk.10.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.10.attention.wv.weight                    -> blk.10.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.10.attention.wo.weight                    -> blk.10.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.10.feed_forward.w1.weight                 -> blk.10.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.10.feed_forward.w2.weight                 -> blk.10.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.10.feed_forward.w3.weight                 -> blk.10.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.10.attention_norm.weight                  -> blk.10.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.10.ffn_norm.weight                        -> blk.10.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.11.attention.wq.weight                    -> blk.11.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.11.attention.wk.weight                    -> blk.11.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.11.attention.wv.weight                    -> blk.11.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.11.attention.wo.weight                    -> blk.11.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.11.feed_forward.w1.weight                 -> blk.11.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.11.feed_forward.w2.weight                 -> blk.11.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.11.feed_forward.w3.weight                 -> blk.11.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.11.attention_norm.weight                  -> blk.11.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.11.ffn_norm.weight                        -> blk.11.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.12.attention.wq.weight                    -> blk.12.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.12.attention.wk.weight                    -> blk.12.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.12.attention.wv.weight                    -> blk.12.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.12.attention.wo.weight                    -> blk.12.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.12.feed_forward.w1.weight                 -> blk.12.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.12.feed_forward.w2.weight                 -> blk.12.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.12.feed_forward.w3.weight                 -> blk.12.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.12.attention_norm.weight                  -> blk.12.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.12.ffn_norm.weight                        -> blk.12.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.13.attention.wq.weight                    -> blk.13.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.13.attention.wk.weight                    -> blk.13.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.13.attention.wv.weight                    -> blk.13.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.13.attention.wo.weight                    -> blk.13.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.13.feed_forward.w1.weight                 -> blk.13.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.13.feed_forward.w2.weight                 -> blk.13.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.13.feed_forward.w3.weight                 -> blk.13.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.13.attention_norm.weight                  -> blk.13.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.13.ffn_norm.weight                        -> blk.13.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.14.attention.wq.weight                    -> blk.14.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.14.attention.wk.weight                    -> blk.14.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.14.attention.wv.weight                    -> blk.14.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.14.attention.wo.weight                    -> blk.14.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.14.feed_forward.w1.weight                 -> blk.14.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.14.feed_forward.w2.weight                 -> blk.14.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.14.feed_forward.w3.weight                 -> blk.14.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.14.attention_norm.weight                  -> blk.14.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.14.ffn_norm.weight                        -> blk.14.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.15.attention.wq.weight                    -> blk.15.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.15.attention.wk.weight                    -> blk.15.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.15.attention.wv.weight                    -> blk.15.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.15.attention.wo.weight                    -> blk.15.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.15.feed_forward.w1.weight                 -> blk.15.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.15.feed_forward.w2.weight                 -> blk.15.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.15.feed_forward.w3.weight                 -> blk.15.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.15.attention_norm.weight                  -> blk.15.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.15.ffn_norm.weight                        -> blk.15.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.16.attention.wq.weight                    -> blk.16.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.16.attention.wk.weight                    -> blk.16.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.16.attention.wv.weight                    -> blk.16.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.16.attention.wo.weight                    -> blk.16.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.16.feed_forward.w1.weight                 -> blk.16.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.16.feed_forward.w2.weight                 -> blk.16.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.16.feed_forward.w3.weight                 -> blk.16.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.16.attention_norm.weight                  -> blk.16.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.16.ffn_norm.weight                        -> blk.16.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.17.attention.wq.weight                    -> blk.17.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.17.attention.wk.weight                    -> blk.17.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.17.attention.wv.weight                    -> blk.17.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.17.attention.wo.weight                    -> blk.17.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.17.feed_forward.w1.weight                 -> blk.17.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.17.feed_forward.w2.weight                 -> blk.17.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.17.feed_forward.w3.weight                 -> blk.17.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.17.attention_norm.weight                  -> blk.17.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.17.ffn_norm.weight                        -> blk.17.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.18.attention.wq.weight                    -> blk.18.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.18.attention.wk.weight                    -> blk.18.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.18.attention.wv.weight                    -> blk.18.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.18.attention.wo.weight                    -> blk.18.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.18.feed_forward.w1.weight                 -> blk.18.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.18.feed_forward.w2.weight                 -> blk.18.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.18.feed_forward.w3.weight                 -> blk.18.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.18.attention_norm.weight                  -> blk.18.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.18.ffn_norm.weight                        -> blk.18.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.19.attention.wq.weight                    -> blk.19.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.19.attention.wk.weight                    -> blk.19.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.19.attention.wv.weight                    -> blk.19.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.19.attention.wo.weight                    -> blk.19.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.19.feed_forward.w1.weight                 -> blk.19.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.19.feed_forward.w2.weight                 -> blk.19.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.19.feed_forward.w3.weight                 -> blk.19.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.19.attention_norm.weight                  -> blk.19.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.19.ffn_norm.weight                        -> blk.19.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.20.attention.wq.weight                    -> blk.20.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.20.attention.wk.weight                    -> blk.20.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.20.attention.wv.weight                    -> blk.20.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.20.attention.wo.weight                    -> blk.20.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.20.feed_forward.w1.weight                 -> blk.20.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.20.feed_forward.w2.weight                 -> blk.20.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.20.feed_forward.w3.weight                 -> blk.20.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.20.attention_norm.weight                  -> blk.20.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.20.ffn_norm.weight                        -> blk.20.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.21.attention.wq.weight                    -> blk.21.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.21.attention.wk.weight                    -> blk.21.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.21.attention.wv.weight                    -> blk.21.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.21.attention.wo.weight                    -> blk.21.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.21.feed_forward.w1.weight                 -> blk.21.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.21.feed_forward.w2.weight                 -> blk.21.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.21.feed_forward.w3.weight                 -> blk.21.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.21.attention_norm.weight                  -> blk.21.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.21.ffn_norm.weight                        -> blk.21.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.22.attention.wq.weight                    -> blk.22.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.22.attention.wk.weight                    -> blk.22.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.22.attention.wv.weight                    -> blk.22.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.22.attention.wo.weight                    -> blk.22.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.22.feed_forward.w1.weight                 -> blk.22.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.22.feed_forward.w2.weight                 -> blk.22.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.22.feed_forward.w3.weight                 -> blk.22.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.22.attention_norm.weight                  -> blk.22.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.22.ffn_norm.weight                        -> blk.22.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.23.attention.wq.weight                    -> blk.23.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.23.attention.wk.weight                    -> blk.23.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.23.attention.wv.weight                    -> blk.23.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.23.attention.wo.weight                    -> blk.23.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.23.feed_forward.w1.weight                 -> blk.23.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.23.feed_forward.w2.weight                 -> blk.23.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.23.feed_forward.w3.weight                 -> blk.23.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.23.attention_norm.weight                  -> blk.23.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.23.ffn_norm.weight                        -> blk.23.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.24.attention.wq.weight                    -> blk.24.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.24.attention.wk.weight                    -> blk.24.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.24.attention.wv.weight                    -> blk.24.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.24.attention.wo.weight                    -> blk.24.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.24.feed_forward.w1.weight                 -> blk.24.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.24.feed_forward.w2.weight                 -> blk.24.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.24.feed_forward.w3.weight                 -> blk.24.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.24.attention_norm.weight                  -> blk.24.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.24.ffn_norm.weight                        -> blk.24.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.25.attention.wq.weight                    -> blk.25.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.25.attention.wk.weight                    -> blk.25.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.25.attention.wv.weight                    -> blk.25.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.25.attention.wo.weight                    -> blk.25.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.25.feed_forward.w1.weight                 -> blk.25.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.25.feed_forward.w2.weight                 -> blk.25.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.25.feed_forward.w3.weight                 -> blk.25.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.25.attention_norm.weight                  -> blk.25.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.25.ffn_norm.weight                        -> blk.25.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.26.attention.wq.weight                    -> blk.26.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.26.attention.wk.weight                    -> blk.26.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.26.attention.wv.weight                    -> blk.26.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.26.attention.wo.weight                    -> blk.26.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.26.feed_forward.w1.weight                 -> blk.26.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.26.feed_forward.w2.weight                 -> blk.26.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.26.feed_forward.w3.weight                 -> blk.26.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.26.attention_norm.weight                  -> blk.26.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.26.ffn_norm.weight                        -> blk.26.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.27.attention.wq.weight                    -> blk.27.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.27.attention.wk.weight                    -> blk.27.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.27.attention.wv.weight                    -> blk.27.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.27.attention.wo.weight                    -> blk.27.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.27.feed_forward.w1.weight                 -> blk.27.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.27.feed_forward.w2.weight                 -> blk.27.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.27.feed_forward.w3.weight                 -> blk.27.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.27.attention_norm.weight                  -> blk.27.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.27.ffn_norm.weight                        -> blk.27.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.28.attention.wq.weight                    -> blk.28.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.28.attention.wk.weight                    -> blk.28.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.28.attention.wv.weight                    -> blk.28.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.28.attention.wo.weight                    -> blk.28.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.28.feed_forward.w1.weight                 -> blk.28.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.28.feed_forward.w2.weight                 -> blk.28.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.28.feed_forward.w3.weight                 -> blk.28.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.28.attention_norm.weight                  -> blk.28.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.28.ffn_norm.weight                        -> blk.28.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.29.attention.wq.weight                    -> blk.29.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.29.attention.wk.weight                    -> blk.29.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.29.attention.wv.weight                    -> blk.29.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.29.attention.wo.weight                    -> blk.29.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.29.feed_forward.w1.weight                 -> blk.29.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.29.feed_forward.w2.weight                 -> blk.29.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.29.feed_forward.w3.weight                 -> blk.29.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.29.attention_norm.weight                  -> blk.29.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.29.ffn_norm.weight                        -> blk.29.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.30.attention.wq.weight                    -> blk.30.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.30.attention.wk.weight                    -> blk.30.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.30.attention.wv.weight                    -> blk.30.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.30.attention.wo.weight                    -> blk.30.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.30.feed_forward.w1.weight                 -> blk.30.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.30.feed_forward.w2.weight                 -> blk.30.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.30.feed_forward.w3.weight                 -> blk.30.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.30.attention_norm.weight                  -> blk.30.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.30.ffn_norm.weight                        -> blk.30.ffn_norm.weight                   | BF16   | [4096]\n",
      "layers.31.attention.wq.weight                    -> blk.31.attn_q.weight                     | BF16   | [4096, 4096]\n",
      "layers.31.attention.wk.weight                    -> blk.31.attn_k.weight                     | BF16   | [4096, 4096]\n",
      "layers.31.attention.wv.weight                    -> blk.31.attn_v.weight                     | BF16   | [4096, 4096]\n",
      "layers.31.attention.wo.weight                    -> blk.31.attn_output.weight                | BF16   | [4096, 4096]\n",
      "layers.31.feed_forward.w1.weight                 -> blk.31.ffn_gate.weight                   | BF16   | [11008, 4096]\n",
      "layers.31.feed_forward.w2.weight                 -> blk.31.ffn_down.weight                   | BF16   | [4096, 11008]\n",
      "layers.31.feed_forward.w3.weight                 -> blk.31.ffn_up.weight                     | BF16   | [11008, 4096]\n",
      "layers.31.attention_norm.weight                  -> blk.31.attn_norm.weight                  | BF16   | [4096]\n",
      "layers.31.ffn_norm.weight                        -> blk.31.ffn_norm.weight                   | BF16   | [4096]\n",
      "skipping tensor rope_freqs\n",
      "Writing models/7B-v2/ggml-model-f16.gguf, format 1\n",
      "gguf: This GGUF file is for Little Endian only\n",
      "gguf: Adding 61249 merge(s).\n",
      "gguf: Setting special token type bos to 1\n",
      "gguf: Setting special token type eos to 2\n",
      "gguf: Setting special token type unk to 0\n",
      "gguf: Setting add_bos_token to True\n",
      "gguf: Setting add_eos_token to False\n",
      "[  1/291] Writing tensor token_embd.weight                      | size  32000 x   4096  | type F16  | T+   1\n",
      "[  2/291] Writing tensor output_norm.weight                     | size   4096           | type F32  | T+   1\n",
      "[  3/291] Writing tensor output.weight                          | size  32000 x   4096  | type F16  | T+   1\n",
      "[  4/291] Writing tensor blk.0.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[  5/291] Writing tensor blk.0.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[  6/291] Writing tensor blk.0.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[  7/291] Writing tensor blk.0.attn_output.weight               | size   4096 x   4096  | type F16  | T+   1\n",
      "[  8/291] Writing tensor blk.0.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   1\n",
      "[  9/291] Writing tensor blk.0.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   1\n",
      "[ 10/291] Writing tensor blk.0.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   1\n",
      "[ 11/291] Writing tensor blk.0.attn_norm.weight                 | size   4096           | type F32  | T+   1\n",
      "[ 12/291] Writing tensor blk.0.ffn_norm.weight                  | size   4096           | type F32  | T+   1\n",
      "[ 13/291] Writing tensor blk.1.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 14/291] Writing tensor blk.1.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 15/291] Writing tensor blk.1.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 16/291] Writing tensor blk.1.attn_output.weight               | size   4096 x   4096  | type F16  | T+   1\n",
      "[ 17/291] Writing tensor blk.1.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 18/291] Writing tensor blk.1.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   2\n",
      "[ 19/291] Writing tensor blk.1.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 20/291] Writing tensor blk.1.attn_norm.weight                 | size   4096           | type F32  | T+   2\n",
      "[ 21/291] Writing tensor blk.1.ffn_norm.weight                  | size   4096           | type F32  | T+   2\n",
      "[ 22/291] Writing tensor blk.2.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 23/291] Writing tensor blk.2.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 24/291] Writing tensor blk.2.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 25/291] Writing tensor blk.2.attn_output.weight               | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 26/291] Writing tensor blk.2.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 27/291] Writing tensor blk.2.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   2\n",
      "[ 28/291] Writing tensor blk.2.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 29/291] Writing tensor blk.2.attn_norm.weight                 | size   4096           | type F32  | T+   2\n",
      "[ 30/291] Writing tensor blk.2.ffn_norm.weight                  | size   4096           | type F32  | T+   2\n",
      "[ 31/291] Writing tensor blk.3.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 32/291] Writing tensor blk.3.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 33/291] Writing tensor blk.3.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 34/291] Writing tensor blk.3.attn_output.weight               | size   4096 x   4096  | type F16  | T+   2\n",
      "[ 35/291] Writing tensor blk.3.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   2\n",
      "[ 36/291] Writing tensor blk.3.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   3\n",
      "[ 37/291] Writing tensor blk.3.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 38/291] Writing tensor blk.3.attn_norm.weight                 | size   4096           | type F32  | T+   3\n",
      "[ 39/291] Writing tensor blk.3.ffn_norm.weight                  | size   4096           | type F32  | T+   3\n",
      "[ 40/291] Writing tensor blk.4.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 41/291] Writing tensor blk.4.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 42/291] Writing tensor blk.4.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 43/291] Writing tensor blk.4.attn_output.weight               | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 44/291] Writing tensor blk.4.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 45/291] Writing tensor blk.4.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   3\n",
      "[ 46/291] Writing tensor blk.4.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 47/291] Writing tensor blk.4.attn_norm.weight                 | size   4096           | type F32  | T+   3\n",
      "[ 48/291] Writing tensor blk.4.ffn_norm.weight                  | size   4096           | type F32  | T+   3\n",
      "[ 49/291] Writing tensor blk.5.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 50/291] Writing tensor blk.5.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 51/291] Writing tensor blk.5.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 52/291] Writing tensor blk.5.attn_output.weight               | size   4096 x   4096  | type F16  | T+   3\n",
      "[ 53/291] Writing tensor blk.5.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   3\n",
      "[ 54/291] Writing tensor blk.5.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   4\n",
      "[ 55/291] Writing tensor blk.5.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 56/291] Writing tensor blk.5.attn_norm.weight                 | size   4096           | type F32  | T+   4\n",
      "[ 57/291] Writing tensor blk.5.ffn_norm.weight                  | size   4096           | type F32  | T+   4\n",
      "[ 58/291] Writing tensor blk.6.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 59/291] Writing tensor blk.6.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 60/291] Writing tensor blk.6.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 61/291] Writing tensor blk.6.attn_output.weight               | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 62/291] Writing tensor blk.6.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 63/291] Writing tensor blk.6.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   4\n",
      "[ 64/291] Writing tensor blk.6.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 65/291] Writing tensor blk.6.attn_norm.weight                 | size   4096           | type F32  | T+   4\n",
      "[ 66/291] Writing tensor blk.6.ffn_norm.weight                  | size   4096           | type F32  | T+   4\n",
      "[ 67/291] Writing tensor blk.7.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 68/291] Writing tensor blk.7.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 69/291] Writing tensor blk.7.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 70/291] Writing tensor blk.7.attn_output.weight               | size   4096 x   4096  | type F16  | T+   4\n",
      "[ 71/291] Writing tensor blk.7.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   4\n",
      "[ 72/291] Writing tensor blk.7.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   5\n",
      "[ 73/291] Writing tensor blk.7.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 74/291] Writing tensor blk.7.attn_norm.weight                 | size   4096           | type F32  | T+   5\n",
      "[ 75/291] Writing tensor blk.7.ffn_norm.weight                  | size   4096           | type F32  | T+   5\n",
      "[ 76/291] Writing tensor blk.8.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 77/291] Writing tensor blk.8.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 78/291] Writing tensor blk.8.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 79/291] Writing tensor blk.8.attn_output.weight               | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 80/291] Writing tensor blk.8.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 81/291] Writing tensor blk.8.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   5\n",
      "[ 82/291] Writing tensor blk.8.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 83/291] Writing tensor blk.8.attn_norm.weight                 | size   4096           | type F32  | T+   5\n",
      "[ 84/291] Writing tensor blk.8.ffn_norm.weight                  | size   4096           | type F32  | T+   5\n",
      "[ 85/291] Writing tensor blk.9.attn_q.weight                    | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 86/291] Writing tensor blk.9.attn_k.weight                    | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 87/291] Writing tensor blk.9.attn_v.weight                    | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 88/291] Writing tensor blk.9.attn_output.weight               | size   4096 x   4096  | type F16  | T+   5\n",
      "[ 89/291] Writing tensor blk.9.ffn_gate.weight                  | size  11008 x   4096  | type F16  | T+   5\n",
      "[ 90/291] Writing tensor blk.9.ffn_down.weight                  | size   4096 x  11008  | type F16  | T+   5\n",
      "[ 91/291] Writing tensor blk.9.ffn_up.weight                    | size  11008 x   4096  | type F16  | T+   6\n",
      "[ 92/291] Writing tensor blk.9.attn_norm.weight                 | size   4096           | type F32  | T+   6\n",
      "[ 93/291] Writing tensor blk.9.ffn_norm.weight                  | size   4096           | type F32  | T+   6\n",
      "[ 94/291] Writing tensor blk.10.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[ 95/291] Writing tensor blk.10.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[ 96/291] Writing tensor blk.10.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[ 97/291] Writing tensor blk.10.attn_output.weight              | size   4096 x   4096  | type F16  | T+   6\n",
      "[ 98/291] Writing tensor blk.10.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   6\n",
      "[ 99/291] Writing tensor blk.10.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   6\n",
      "[100/291] Writing tensor blk.10.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   6\n",
      "[101/291] Writing tensor blk.10.attn_norm.weight                | size   4096           | type F32  | T+   6\n",
      "[102/291] Writing tensor blk.10.ffn_norm.weight                 | size   4096           | type F32  | T+   6\n",
      "[103/291] Writing tensor blk.11.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[104/291] Writing tensor blk.11.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[105/291] Writing tensor blk.11.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   6\n",
      "[106/291] Writing tensor blk.11.attn_output.weight              | size   4096 x   4096  | type F16  | T+   6\n",
      "[107/291] Writing tensor blk.11.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   6\n",
      "[108/291] Writing tensor blk.11.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   6\n",
      "[109/291] Writing tensor blk.11.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   7\n",
      "[110/291] Writing tensor blk.11.attn_norm.weight                | size   4096           | type F32  | T+   7\n",
      "[111/291] Writing tensor blk.11.ffn_norm.weight                 | size   4096           | type F32  | T+   7\n",
      "[112/291] Writing tensor blk.12.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[113/291] Writing tensor blk.12.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[114/291] Writing tensor blk.12.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[115/291] Writing tensor blk.12.attn_output.weight              | size   4096 x   4096  | type F16  | T+   7\n",
      "[116/291] Writing tensor blk.12.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   7\n",
      "[117/291] Writing tensor blk.12.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   7\n",
      "[118/291] Writing tensor blk.12.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   7\n",
      "[119/291] Writing tensor blk.12.attn_norm.weight                | size   4096           | type F32  | T+   7\n",
      "[120/291] Writing tensor blk.12.ffn_norm.weight                 | size   4096           | type F32  | T+   7\n",
      "[121/291] Writing tensor blk.13.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[122/291] Writing tensor blk.13.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[123/291] Writing tensor blk.13.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[124/291] Writing tensor blk.13.attn_output.weight              | size   4096 x   4096  | type F16  | T+   7\n",
      "[125/291] Writing tensor blk.13.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   7\n",
      "[126/291] Writing tensor blk.13.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   7\n",
      "[127/291] Writing tensor blk.13.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   7\n",
      "[128/291] Writing tensor blk.13.attn_norm.weight                | size   4096           | type F32  | T+   7\n",
      "[129/291] Writing tensor blk.13.ffn_norm.weight                 | size   4096           | type F32  | T+   7\n",
      "[130/291] Writing tensor blk.14.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[131/291] Writing tensor blk.14.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   7\n",
      "[132/291] Writing tensor blk.14.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[133/291] Writing tensor blk.14.attn_output.weight              | size   4096 x   4096  | type F16  | T+   8\n",
      "[134/291] Writing tensor blk.14.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   8\n",
      "[135/291] Writing tensor blk.14.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   8\n",
      "[136/291] Writing tensor blk.14.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   8\n",
      "[137/291] Writing tensor blk.14.attn_norm.weight                | size   4096           | type F32  | T+   8\n",
      "[138/291] Writing tensor blk.14.ffn_norm.weight                 | size   4096           | type F32  | T+   8\n",
      "[139/291] Writing tensor blk.15.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[140/291] Writing tensor blk.15.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[141/291] Writing tensor blk.15.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   8\n",
      "[142/291] Writing tensor blk.15.attn_output.weight              | size   4096 x   4096  | type F16  | T+   8\n",
      "[143/291] Writing tensor blk.15.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   8\n",
      "[144/291] Writing tensor blk.15.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   8\n",
      "[145/291] Writing tensor blk.15.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   8\n",
      "[146/291] Writing tensor blk.15.attn_norm.weight                | size   4096           | type F32  | T+   9\n",
      "[147/291] Writing tensor blk.15.ffn_norm.weight                 | size   4096           | type F32  | T+   9\n",
      "[148/291] Writing tensor blk.16.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[149/291] Writing tensor blk.16.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[150/291] Writing tensor blk.16.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[151/291] Writing tensor blk.16.attn_output.weight              | size   4096 x   4096  | type F16  | T+   9\n",
      "[152/291] Writing tensor blk.16.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   9\n",
      "[153/291] Writing tensor blk.16.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   9\n",
      "[154/291] Writing tensor blk.16.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   9\n",
      "[155/291] Writing tensor blk.16.attn_norm.weight                | size   4096           | type F32  | T+   9\n",
      "[156/291] Writing tensor blk.16.ffn_norm.weight                 | size   4096           | type F32  | T+   9\n",
      "[157/291] Writing tensor blk.17.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[158/291] Writing tensor blk.17.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[159/291] Writing tensor blk.17.attn_v.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[160/291] Writing tensor blk.17.attn_output.weight              | size   4096 x   4096  | type F16  | T+   9\n",
      "[161/291] Writing tensor blk.17.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+   9\n",
      "[162/291] Writing tensor blk.17.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+   9\n",
      "[163/291] Writing tensor blk.17.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+   9\n",
      "[164/291] Writing tensor blk.17.attn_norm.weight                | size   4096           | type F32  | T+   9\n",
      "[165/291] Writing tensor blk.17.ffn_norm.weight                 | size   4096           | type F32  | T+   9\n",
      "[166/291] Writing tensor blk.18.attn_q.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[167/291] Writing tensor blk.18.attn_k.weight                   | size   4096 x   4096  | type F16  | T+   9\n",
      "[168/291] Writing tensor blk.18.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[169/291] Writing tensor blk.18.attn_output.weight              | size   4096 x   4096  | type F16  | T+  10\n",
      "[170/291] Writing tensor blk.18.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  10\n",
      "[171/291] Writing tensor blk.18.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  10\n",
      "[172/291] Writing tensor blk.18.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  10\n",
      "[173/291] Writing tensor blk.18.attn_norm.weight                | size   4096           | type F32  | T+  10\n",
      "[174/291] Writing tensor blk.18.ffn_norm.weight                 | size   4096           | type F32  | T+  10\n",
      "[175/291] Writing tensor blk.19.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[176/291] Writing tensor blk.19.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[177/291] Writing tensor blk.19.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  10\n",
      "[178/291] Writing tensor blk.19.attn_output.weight              | size   4096 x   4096  | type F16  | T+  10\n",
      "[179/291] Writing tensor blk.19.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  10\n",
      "[180/291] Writing tensor blk.19.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  11\n",
      "[181/291] Writing tensor blk.19.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  11\n",
      "[182/291] Writing tensor blk.19.attn_norm.weight                | size   4096           | type F32  | T+  11\n",
      "[183/291] Writing tensor blk.19.ffn_norm.weight                 | size   4096           | type F32  | T+  11\n",
      "[184/291] Writing tensor blk.20.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[185/291] Writing tensor blk.20.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[186/291] Writing tensor blk.20.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[187/291] Writing tensor blk.20.attn_output.weight              | size   4096 x   4096  | type F16  | T+  11\n",
      "[188/291] Writing tensor blk.20.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  11\n",
      "[189/291] Writing tensor blk.20.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  11\n",
      "[190/291] Writing tensor blk.20.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  11\n",
      "[191/291] Writing tensor blk.20.attn_norm.weight                | size   4096           | type F32  | T+  11\n",
      "[192/291] Writing tensor blk.20.ffn_norm.weight                 | size   4096           | type F32  | T+  11\n",
      "[193/291] Writing tensor blk.21.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[194/291] Writing tensor blk.21.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[195/291] Writing tensor blk.21.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  11\n",
      "[196/291] Writing tensor blk.21.attn_output.weight              | size   4096 x   4096  | type F16  | T+  11\n",
      "[197/291] Writing tensor blk.21.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  12\n",
      "[198/291] Writing tensor blk.21.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  12\n",
      "[199/291] Writing tensor blk.21.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  12\n",
      "[200/291] Writing tensor blk.21.attn_norm.weight                | size   4096           | type F32  | T+  12\n",
      "[201/291] Writing tensor blk.21.ffn_norm.weight                 | size   4096           | type F32  | T+  12\n",
      "[202/291] Writing tensor blk.22.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[203/291] Writing tensor blk.22.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[204/291] Writing tensor blk.22.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  12\n",
      "[205/291] Writing tensor blk.22.attn_output.weight              | size   4096 x   4096  | type F16  | T+  12\n",
      "[206/291] Writing tensor blk.22.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  12\n",
      "[207/291] Writing tensor blk.22.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  12\n",
      "[208/291] Writing tensor blk.22.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  12\n",
      "[209/291] Writing tensor blk.22.attn_norm.weight                | size   4096           | type F32  | T+  13\n",
      "[210/291] Writing tensor blk.22.ffn_norm.weight                 | size   4096           | type F32  | T+  13\n",
      "[211/291] Writing tensor blk.23.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[212/291] Writing tensor blk.23.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[213/291] Writing tensor blk.23.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[214/291] Writing tensor blk.23.attn_output.weight              | size   4096 x   4096  | type F16  | T+  13\n",
      "[215/291] Writing tensor blk.23.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  13\n",
      "[216/291] Writing tensor blk.23.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  13\n",
      "[217/291] Writing tensor blk.23.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  13\n",
      "[218/291] Writing tensor blk.23.attn_norm.weight                | size   4096           | type F32  | T+  13\n",
      "[219/291] Writing tensor blk.23.ffn_norm.weight                 | size   4096           | type F32  | T+  13\n",
      "[220/291] Writing tensor blk.24.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[221/291] Writing tensor blk.24.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[222/291] Writing tensor blk.24.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  13\n",
      "[223/291] Writing tensor blk.24.attn_output.weight              | size   4096 x   4096  | type F16  | T+  13\n",
      "[224/291] Writing tensor blk.24.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  13\n",
      "[225/291] Writing tensor blk.24.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  13\n",
      "[226/291] Writing tensor blk.24.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  14\n",
      "[227/291] Writing tensor blk.24.attn_norm.weight                | size   4096           | type F32  | T+  14\n",
      "[228/291] Writing tensor blk.24.ffn_norm.weight                 | size   4096           | type F32  | T+  14\n",
      "[229/291] Writing tensor blk.25.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  14\n",
      "[230/291] Writing tensor blk.25.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  14\n",
      "[231/291] Writing tensor blk.25.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  14\n",
      "[232/291] Writing tensor blk.25.attn_output.weight              | size   4096 x   4096  | type F16  | T+  14\n",
      "[233/291] Writing tensor blk.25.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  14\n",
      "[234/291] Writing tensor blk.25.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  14\n",
      "[235/291] Writing tensor blk.25.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  14\n",
      "[236/291] Writing tensor blk.25.attn_norm.weight                | size   4096           | type F32  | T+  14\n",
      "[237/291] Writing tensor blk.25.ffn_norm.weight                 | size   4096           | type F32  | T+  14\n",
      "[238/291] Writing tensor blk.26.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  14\n",
      "[239/291] Writing tensor blk.26.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  14\n",
      "[240/291] Writing tensor blk.26.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  14\n",
      "[241/291] Writing tensor blk.26.attn_output.weight              | size   4096 x   4096  | type F16  | T+  14\n",
      "[242/291] Writing tensor blk.26.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  14\n",
      "[243/291] Writing tensor blk.26.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  14\n",
      "[244/291] Writing tensor blk.26.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  15\n",
      "[245/291] Writing tensor blk.26.attn_norm.weight                | size   4096           | type F32  | T+  15\n",
      "[246/291] Writing tensor blk.26.ffn_norm.weight                 | size   4096           | type F32  | T+  15\n",
      "[247/291] Writing tensor blk.27.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  15\n",
      "[248/291] Writing tensor blk.27.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  15\n",
      "[249/291] Writing tensor blk.27.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  15\n",
      "[250/291] Writing tensor blk.27.attn_output.weight              | size   4096 x   4096  | type F16  | T+  15\n",
      "[251/291] Writing tensor blk.27.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  15\n",
      "[252/291] Writing tensor blk.27.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  15\n",
      "[253/291] Writing tensor blk.27.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  15\n",
      "[254/291] Writing tensor blk.27.attn_norm.weight                | size   4096           | type F32  | T+  15\n",
      "[255/291] Writing tensor blk.27.ffn_norm.weight                 | size   4096           | type F32  | T+  15\n",
      "[256/291] Writing tensor blk.28.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  15\n",
      "[257/291] Writing tensor blk.28.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  15\n",
      "[258/291] Writing tensor blk.28.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  15\n",
      "[259/291] Writing tensor blk.28.attn_output.weight              | size   4096 x   4096  | type F16  | T+  15\n",
      "[260/291] Writing tensor blk.28.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  16\n",
      "[261/291] Writing tensor blk.28.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  17\n",
      "[262/291] Writing tensor blk.28.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  17\n",
      "[263/291] Writing tensor blk.28.attn_norm.weight                | size   4096           | type F32  | T+  17\n",
      "[264/291] Writing tensor blk.28.ffn_norm.weight                 | size   4096           | type F32  | T+  18\n",
      "[265/291] Writing tensor blk.29.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  18\n",
      "[266/291] Writing tensor blk.29.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  18\n",
      "[267/291] Writing tensor blk.29.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  18\n",
      "[268/291] Writing tensor blk.29.attn_output.weight              | size   4096 x   4096  | type F16  | T+  18\n",
      "[269/291] Writing tensor blk.29.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  18\n",
      "[270/291] Writing tensor blk.29.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  19\n",
      "[271/291] Writing tensor blk.29.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  19\n",
      "[272/291] Writing tensor blk.29.attn_norm.weight                | size   4096           | type F32  | T+  19\n",
      "[273/291] Writing tensor blk.29.ffn_norm.weight                 | size   4096           | type F32  | T+  19\n",
      "[274/291] Writing tensor blk.30.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  19\n",
      "[275/291] Writing tensor blk.30.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  19\n",
      "[276/291] Writing tensor blk.30.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  19\n",
      "[277/291] Writing tensor blk.30.attn_output.weight              | size   4096 x   4096  | type F16  | T+  19\n",
      "[278/291] Writing tensor blk.30.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  19\n",
      "[279/291] Writing tensor blk.30.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  19\n",
      "[280/291] Writing tensor blk.30.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  19\n",
      "[281/291] Writing tensor blk.30.attn_norm.weight                | size   4096           | type F32  | T+  19\n",
      "[282/291] Writing tensor blk.30.ffn_norm.weight                 | size   4096           | type F32  | T+  19\n",
      "[283/291] Writing tensor blk.31.attn_q.weight                   | size   4096 x   4096  | type F16  | T+  19\n",
      "[284/291] Writing tensor blk.31.attn_k.weight                   | size   4096 x   4096  | type F16  | T+  19\n",
      "[285/291] Writing tensor blk.31.attn_v.weight                   | size   4096 x   4096  | type F16  | T+  19\n",
      "[286/291] Writing tensor blk.31.attn_output.weight              | size   4096 x   4096  | type F16  | T+  19\n",
      "[287/291] Writing tensor blk.31.ffn_gate.weight                 | size  11008 x   4096  | type F16  | T+  19\n",
      "[288/291] Writing tensor blk.31.ffn_down.weight                 | size   4096 x  11008  | type F16  | T+  20\n",
      "[289/291] Writing tensor blk.31.ffn_up.weight                   | size  11008 x   4096  | type F16  | T+  20\n",
      "[290/291] Writing tensor blk.31.attn_norm.weight                | size   4096           | type F32  | T+  20\n",
      "[291/291] Writing tensor blk.31.ffn_norm.weight                 | size   4096           | type F32  | T+  20\n",
      "Wrote models/7B-v2/ggml-model-f16.gguf\n",
      "Loading model file models/13B-v2/consolidated.00.pth\n",
      "Loading model file models/13B-v2/consolidated.01.pth\n",
      "params = Params(n_vocab=32000, n_embd=5120, n_layer=40, n_ctx=4096, n_ff=13824, n_head=40, n_head_kv=40, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('models/13B-v2'))\n",
      "32000 32000\n",
      "Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>\n",
      "Special vocab info: <SpecialVocab with 61249 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0}, add special tokens {'bos': True, 'eos': False}>\n",
      "tok_embeddings.weight                            -> token_embd.weight                        | BF16   | [32000, 5120]\n",
      "norm.weight                                      -> output_norm.weight                       | BF16   | [5120]\n",
      "output.weight                                    -> output.weight                            | BF16   | [32000, 5120]\n",
      "layers.0.attention.wq.weight                     -> blk.0.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.0.attention.wk.weight                     -> blk.0.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.0.attention.wv.weight                     -> blk.0.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.0.attention.wo.weight                     -> blk.0.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.0.feed_forward.w1.weight                  -> blk.0.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.0.feed_forward.w2.weight                  -> blk.0.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.0.feed_forward.w3.weight                  -> blk.0.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.0.attention_norm.weight                   -> blk.0.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.0.ffn_norm.weight                         -> blk.0.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.1.attention.wq.weight                     -> blk.1.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.1.attention.wk.weight                     -> blk.1.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.1.attention.wv.weight                     -> blk.1.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.1.attention.wo.weight                     -> blk.1.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.1.feed_forward.w1.weight                  -> blk.1.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.1.feed_forward.w2.weight                  -> blk.1.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.1.feed_forward.w3.weight                  -> blk.1.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.1.attention_norm.weight                   -> blk.1.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.1.ffn_norm.weight                         -> blk.1.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.2.attention.wq.weight                     -> blk.2.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.2.attention.wk.weight                     -> blk.2.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.2.attention.wv.weight                     -> blk.2.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.2.attention.wo.weight                     -> blk.2.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.2.feed_forward.w1.weight                  -> blk.2.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.2.feed_forward.w2.weight                  -> blk.2.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.2.feed_forward.w3.weight                  -> blk.2.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.2.attention_norm.weight                   -> blk.2.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.2.ffn_norm.weight                         -> blk.2.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.3.attention.wq.weight                     -> blk.3.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.3.attention.wk.weight                     -> blk.3.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.3.attention.wv.weight                     -> blk.3.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.3.attention.wo.weight                     -> blk.3.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.3.feed_forward.w1.weight                  -> blk.3.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.3.feed_forward.w2.weight                  -> blk.3.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.3.feed_forward.w3.weight                  -> blk.3.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.3.attention_norm.weight                   -> blk.3.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.3.ffn_norm.weight                         -> blk.3.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.4.attention.wq.weight                     -> blk.4.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.4.attention.wk.weight                     -> blk.4.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.4.attention.wv.weight                     -> blk.4.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.4.attention.wo.weight                     -> blk.4.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.4.feed_forward.w1.weight                  -> blk.4.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.4.feed_forward.w2.weight                  -> blk.4.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.4.feed_forward.w3.weight                  -> blk.4.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.4.attention_norm.weight                   -> blk.4.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.4.ffn_norm.weight                         -> blk.4.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.5.attention.wq.weight                     -> blk.5.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.5.attention.wk.weight                     -> blk.5.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.5.attention.wv.weight                     -> blk.5.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.5.attention.wo.weight                     -> blk.5.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.5.feed_forward.w1.weight                  -> blk.5.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.5.feed_forward.w2.weight                  -> blk.5.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.5.feed_forward.w3.weight                  -> blk.5.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.5.attention_norm.weight                   -> blk.5.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.5.ffn_norm.weight                         -> blk.5.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.6.attention.wq.weight                     -> blk.6.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.6.attention.wk.weight                     -> blk.6.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.6.attention.wv.weight                     -> blk.6.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.6.attention.wo.weight                     -> blk.6.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.6.feed_forward.w1.weight                  -> blk.6.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.6.feed_forward.w2.weight                  -> blk.6.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.6.feed_forward.w3.weight                  -> blk.6.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.6.attention_norm.weight                   -> blk.6.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.6.ffn_norm.weight                         -> blk.6.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.7.attention.wq.weight                     -> blk.7.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.7.attention.wk.weight                     -> blk.7.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.7.attention.wv.weight                     -> blk.7.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.7.attention.wo.weight                     -> blk.7.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.7.feed_forward.w1.weight                  -> blk.7.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.7.feed_forward.w2.weight                  -> blk.7.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.7.feed_forward.w3.weight                  -> blk.7.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.7.attention_norm.weight                   -> blk.7.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.7.ffn_norm.weight                         -> blk.7.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.8.attention.wq.weight                     -> blk.8.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.8.attention.wk.weight                     -> blk.8.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.8.attention.wv.weight                     -> blk.8.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.8.attention.wo.weight                     -> blk.8.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.8.feed_forward.w1.weight                  -> blk.8.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.8.feed_forward.w2.weight                  -> blk.8.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.8.feed_forward.w3.weight                  -> blk.8.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.8.attention_norm.weight                   -> blk.8.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.8.ffn_norm.weight                         -> blk.8.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.9.attention.wq.weight                     -> blk.9.attn_q.weight                      | BF16   | [5120, 5120]\n",
      "layers.9.attention.wk.weight                     -> blk.9.attn_k.weight                      | BF16   | [5120, 5120]\n",
      "layers.9.attention.wv.weight                     -> blk.9.attn_v.weight                      | BF16   | [5120, 5120]\n",
      "layers.9.attention.wo.weight                     -> blk.9.attn_output.weight                 | BF16   | [5120, 5120]\n",
      "layers.9.feed_forward.w1.weight                  -> blk.9.ffn_gate.weight                    | BF16   | [13824, 5120]\n",
      "layers.9.feed_forward.w2.weight                  -> blk.9.ffn_down.weight                    | BF16   | [5120, 13824]\n",
      "layers.9.feed_forward.w3.weight                  -> blk.9.ffn_up.weight                      | BF16   | [13824, 5120]\n",
      "layers.9.attention_norm.weight                   -> blk.9.attn_norm.weight                   | BF16   | [5120]\n",
      "layers.9.ffn_norm.weight                         -> blk.9.ffn_norm.weight                    | BF16   | [5120]\n",
      "layers.10.attention.wq.weight                    -> blk.10.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.10.attention.wk.weight                    -> blk.10.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.10.attention.wv.weight                    -> blk.10.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.10.attention.wo.weight                    -> blk.10.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.10.feed_forward.w1.weight                 -> blk.10.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.10.feed_forward.w2.weight                 -> blk.10.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.10.feed_forward.w3.weight                 -> blk.10.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.10.attention_norm.weight                  -> blk.10.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.10.ffn_norm.weight                        -> blk.10.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.11.attention.wq.weight                    -> blk.11.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.11.attention.wk.weight                    -> blk.11.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.11.attention.wv.weight                    -> blk.11.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.11.attention.wo.weight                    -> blk.11.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.11.feed_forward.w1.weight                 -> blk.11.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.11.feed_forward.w2.weight                 -> blk.11.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.11.feed_forward.w3.weight                 -> blk.11.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.11.attention_norm.weight                  -> blk.11.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.11.ffn_norm.weight                        -> blk.11.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.12.attention.wq.weight                    -> blk.12.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.12.attention.wk.weight                    -> blk.12.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.12.attention.wv.weight                    -> blk.12.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.12.attention.wo.weight                    -> blk.12.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.12.feed_forward.w1.weight                 -> blk.12.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.12.feed_forward.w2.weight                 -> blk.12.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.12.feed_forward.w3.weight                 -> blk.12.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.12.attention_norm.weight                  -> blk.12.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.12.ffn_norm.weight                        -> blk.12.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.13.attention.wq.weight                    -> blk.13.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.13.attention.wk.weight                    -> blk.13.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.13.attention.wv.weight                    -> blk.13.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.13.attention.wo.weight                    -> blk.13.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.13.feed_forward.w1.weight                 -> blk.13.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.13.feed_forward.w2.weight                 -> blk.13.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.13.feed_forward.w3.weight                 -> blk.13.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.13.attention_norm.weight                  -> blk.13.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.13.ffn_norm.weight                        -> blk.13.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.14.attention.wq.weight                    -> blk.14.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.14.attention.wk.weight                    -> blk.14.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.14.attention.wv.weight                    -> blk.14.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.14.attention.wo.weight                    -> blk.14.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.14.feed_forward.w1.weight                 -> blk.14.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.14.feed_forward.w2.weight                 -> blk.14.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.14.feed_forward.w3.weight                 -> blk.14.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.14.attention_norm.weight                  -> blk.14.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.14.ffn_norm.weight                        -> blk.14.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.15.attention.wq.weight                    -> blk.15.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.15.attention.wk.weight                    -> blk.15.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.15.attention.wv.weight                    -> blk.15.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.15.attention.wo.weight                    -> blk.15.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.15.feed_forward.w1.weight                 -> blk.15.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.15.feed_forward.w2.weight                 -> blk.15.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.15.feed_forward.w3.weight                 -> blk.15.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.15.attention_norm.weight                  -> blk.15.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.15.ffn_norm.weight                        -> blk.15.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.16.attention.wq.weight                    -> blk.16.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.16.attention.wk.weight                    -> blk.16.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.16.attention.wv.weight                    -> blk.16.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.16.attention.wo.weight                    -> blk.16.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.16.feed_forward.w1.weight                 -> blk.16.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.16.feed_forward.w2.weight                 -> blk.16.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.16.feed_forward.w3.weight                 -> blk.16.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.16.attention_norm.weight                  -> blk.16.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.16.ffn_norm.weight                        -> blk.16.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.17.attention.wq.weight                    -> blk.17.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.17.attention.wk.weight                    -> blk.17.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.17.attention.wv.weight                    -> blk.17.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.17.attention.wo.weight                    -> blk.17.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.17.feed_forward.w1.weight                 -> blk.17.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.17.feed_forward.w2.weight                 -> blk.17.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.17.feed_forward.w3.weight                 -> blk.17.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.17.attention_norm.weight                  -> blk.17.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.17.ffn_norm.weight                        -> blk.17.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.18.attention.wq.weight                    -> blk.18.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.18.attention.wk.weight                    -> blk.18.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.18.attention.wv.weight                    -> blk.18.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.18.attention.wo.weight                    -> blk.18.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.18.feed_forward.w1.weight                 -> blk.18.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.18.feed_forward.w2.weight                 -> blk.18.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.18.feed_forward.w3.weight                 -> blk.18.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.18.attention_norm.weight                  -> blk.18.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.18.ffn_norm.weight                        -> blk.18.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.19.attention.wq.weight                    -> blk.19.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.19.attention.wk.weight                    -> blk.19.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.19.attention.wv.weight                    -> blk.19.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.19.attention.wo.weight                    -> blk.19.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.19.feed_forward.w1.weight                 -> blk.19.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.19.feed_forward.w2.weight                 -> blk.19.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.19.feed_forward.w3.weight                 -> blk.19.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.19.attention_norm.weight                  -> blk.19.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.19.ffn_norm.weight                        -> blk.19.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.20.attention.wq.weight                    -> blk.20.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.20.attention.wk.weight                    -> blk.20.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.20.attention.wv.weight                    -> blk.20.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.20.attention.wo.weight                    -> blk.20.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.20.feed_forward.w1.weight                 -> blk.20.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.20.feed_forward.w2.weight                 -> blk.20.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.20.feed_forward.w3.weight                 -> blk.20.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.20.attention_norm.weight                  -> blk.20.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.20.ffn_norm.weight                        -> blk.20.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.21.attention.wq.weight                    -> blk.21.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.21.attention.wk.weight                    -> blk.21.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.21.attention.wv.weight                    -> blk.21.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.21.attention.wo.weight                    -> blk.21.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.21.feed_forward.w1.weight                 -> blk.21.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.21.feed_forward.w2.weight                 -> blk.21.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.21.feed_forward.w3.weight                 -> blk.21.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.21.attention_norm.weight                  -> blk.21.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.21.ffn_norm.weight                        -> blk.21.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.22.attention.wq.weight                    -> blk.22.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.22.attention.wk.weight                    -> blk.22.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.22.attention.wv.weight                    -> blk.22.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.22.attention.wo.weight                    -> blk.22.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.22.feed_forward.w1.weight                 -> blk.22.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.22.feed_forward.w2.weight                 -> blk.22.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.22.feed_forward.w3.weight                 -> blk.22.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.22.attention_norm.weight                  -> blk.22.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.22.ffn_norm.weight                        -> blk.22.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.23.attention.wq.weight                    -> blk.23.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.23.attention.wk.weight                    -> blk.23.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.23.attention.wv.weight                    -> blk.23.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.23.attention.wo.weight                    -> blk.23.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.23.feed_forward.w1.weight                 -> blk.23.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.23.feed_forward.w2.weight                 -> blk.23.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.23.feed_forward.w3.weight                 -> blk.23.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.23.attention_norm.weight                  -> blk.23.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.23.ffn_norm.weight                        -> blk.23.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.24.attention.wq.weight                    -> blk.24.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.24.attention.wk.weight                    -> blk.24.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.24.attention.wv.weight                    -> blk.24.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.24.attention.wo.weight                    -> blk.24.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.24.feed_forward.w1.weight                 -> blk.24.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.24.feed_forward.w2.weight                 -> blk.24.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.24.feed_forward.w3.weight                 -> blk.24.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.24.attention_norm.weight                  -> blk.24.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.24.ffn_norm.weight                        -> blk.24.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.25.attention.wq.weight                    -> blk.25.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.25.attention.wk.weight                    -> blk.25.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.25.attention.wv.weight                    -> blk.25.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.25.attention.wo.weight                    -> blk.25.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.25.feed_forward.w1.weight                 -> blk.25.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.25.feed_forward.w2.weight                 -> blk.25.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.25.feed_forward.w3.weight                 -> blk.25.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.25.attention_norm.weight                  -> blk.25.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.25.ffn_norm.weight                        -> blk.25.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.26.attention.wq.weight                    -> blk.26.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.26.attention.wk.weight                    -> blk.26.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.26.attention.wv.weight                    -> blk.26.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.26.attention.wo.weight                    -> blk.26.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.26.feed_forward.w1.weight                 -> blk.26.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.26.feed_forward.w2.weight                 -> blk.26.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.26.feed_forward.w3.weight                 -> blk.26.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.26.attention_norm.weight                  -> blk.26.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.26.ffn_norm.weight                        -> blk.26.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.27.attention.wq.weight                    -> blk.27.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.27.attention.wk.weight                    -> blk.27.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.27.attention.wv.weight                    -> blk.27.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.27.attention.wo.weight                    -> blk.27.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.27.feed_forward.w1.weight                 -> blk.27.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.27.feed_forward.w2.weight                 -> blk.27.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.27.feed_forward.w3.weight                 -> blk.27.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.27.attention_norm.weight                  -> blk.27.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.27.ffn_norm.weight                        -> blk.27.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.28.attention.wq.weight                    -> blk.28.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.28.attention.wk.weight                    -> blk.28.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.28.attention.wv.weight                    -> blk.28.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.28.attention.wo.weight                    -> blk.28.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.28.feed_forward.w1.weight                 -> blk.28.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.28.feed_forward.w2.weight                 -> blk.28.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.28.feed_forward.w3.weight                 -> blk.28.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.28.attention_norm.weight                  -> blk.28.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.28.ffn_norm.weight                        -> blk.28.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.29.attention.wq.weight                    -> blk.29.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.29.attention.wk.weight                    -> blk.29.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.29.attention.wv.weight                    -> blk.29.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.29.attention.wo.weight                    -> blk.29.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.29.feed_forward.w1.weight                 -> blk.29.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.29.feed_forward.w2.weight                 -> blk.29.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.29.feed_forward.w3.weight                 -> blk.29.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.29.attention_norm.weight                  -> blk.29.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.29.ffn_norm.weight                        -> blk.29.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.30.attention.wq.weight                    -> blk.30.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.30.attention.wk.weight                    -> blk.30.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.30.attention.wv.weight                    -> blk.30.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.30.attention.wo.weight                    -> blk.30.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.30.feed_forward.w1.weight                 -> blk.30.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.30.feed_forward.w2.weight                 -> blk.30.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.30.feed_forward.w3.weight                 -> blk.30.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.30.attention_norm.weight                  -> blk.30.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.30.ffn_norm.weight                        -> blk.30.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.31.attention.wq.weight                    -> blk.31.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.31.attention.wk.weight                    -> blk.31.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.31.attention.wv.weight                    -> blk.31.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.31.attention.wo.weight                    -> blk.31.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.31.feed_forward.w1.weight                 -> blk.31.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.31.feed_forward.w2.weight                 -> blk.31.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.31.feed_forward.w3.weight                 -> blk.31.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.31.attention_norm.weight                  -> blk.31.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.31.ffn_norm.weight                        -> blk.31.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.32.attention.wq.weight                    -> blk.32.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.32.attention.wk.weight                    -> blk.32.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.32.attention.wv.weight                    -> blk.32.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.32.attention.wo.weight                    -> blk.32.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.32.feed_forward.w1.weight                 -> blk.32.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.32.feed_forward.w2.weight                 -> blk.32.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.32.feed_forward.w3.weight                 -> blk.32.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.32.attention_norm.weight                  -> blk.32.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.32.ffn_norm.weight                        -> blk.32.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.33.attention.wq.weight                    -> blk.33.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.33.attention.wk.weight                    -> blk.33.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.33.attention.wv.weight                    -> blk.33.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.33.attention.wo.weight                    -> blk.33.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.33.feed_forward.w1.weight                 -> blk.33.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.33.feed_forward.w2.weight                 -> blk.33.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.33.feed_forward.w3.weight                 -> blk.33.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.33.attention_norm.weight                  -> blk.33.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.33.ffn_norm.weight                        -> blk.33.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.34.attention.wq.weight                    -> blk.34.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.34.attention.wk.weight                    -> blk.34.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.34.attention.wv.weight                    -> blk.34.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.34.attention.wo.weight                    -> blk.34.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.34.feed_forward.w1.weight                 -> blk.34.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.34.feed_forward.w2.weight                 -> blk.34.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.34.feed_forward.w3.weight                 -> blk.34.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.34.attention_norm.weight                  -> blk.34.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.34.ffn_norm.weight                        -> blk.34.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.35.attention.wq.weight                    -> blk.35.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.35.attention.wk.weight                    -> blk.35.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.35.attention.wv.weight                    -> blk.35.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.35.attention.wo.weight                    -> blk.35.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.35.feed_forward.w1.weight                 -> blk.35.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.35.feed_forward.w2.weight                 -> blk.35.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.35.feed_forward.w3.weight                 -> blk.35.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.35.attention_norm.weight                  -> blk.35.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.35.ffn_norm.weight                        -> blk.35.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.36.attention.wq.weight                    -> blk.36.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.36.attention.wk.weight                    -> blk.36.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.36.attention.wv.weight                    -> blk.36.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.36.attention.wo.weight                    -> blk.36.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.36.feed_forward.w1.weight                 -> blk.36.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.36.feed_forward.w2.weight                 -> blk.36.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.36.feed_forward.w3.weight                 -> blk.36.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.36.attention_norm.weight                  -> blk.36.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.36.ffn_norm.weight                        -> blk.36.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.37.attention.wq.weight                    -> blk.37.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.37.attention.wk.weight                    -> blk.37.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.37.attention.wv.weight                    -> blk.37.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.37.attention.wo.weight                    -> blk.37.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.37.feed_forward.w1.weight                 -> blk.37.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.37.feed_forward.w2.weight                 -> blk.37.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.37.feed_forward.w3.weight                 -> blk.37.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.37.attention_norm.weight                  -> blk.37.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.37.ffn_norm.weight                        -> blk.37.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.38.attention.wq.weight                    -> blk.38.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.38.attention.wk.weight                    -> blk.38.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.38.attention.wv.weight                    -> blk.38.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.38.attention.wo.weight                    -> blk.38.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.38.feed_forward.w1.weight                 -> blk.38.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.38.feed_forward.w2.weight                 -> blk.38.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.38.feed_forward.w3.weight                 -> blk.38.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.38.attention_norm.weight                  -> blk.38.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.38.ffn_norm.weight                        -> blk.38.ffn_norm.weight                   | BF16   | [5120]\n",
      "layers.39.attention.wq.weight                    -> blk.39.attn_q.weight                     | BF16   | [5120, 5120]\n",
      "layers.39.attention.wk.weight                    -> blk.39.attn_k.weight                     | BF16   | [5120, 5120]\n",
      "layers.39.attention.wv.weight                    -> blk.39.attn_v.weight                     | BF16   | [5120, 5120]\n",
      "layers.39.attention.wo.weight                    -> blk.39.attn_output.weight                | BF16   | [5120, 5120]\n",
      "layers.39.feed_forward.w1.weight                 -> blk.39.ffn_gate.weight                   | BF16   | [13824, 5120]\n",
      "layers.39.feed_forward.w2.weight                 -> blk.39.ffn_down.weight                   | BF16   | [5120, 13824]\n",
      "layers.39.feed_forward.w3.weight                 -> blk.39.ffn_up.weight                     | BF16   | [13824, 5120]\n",
      "layers.39.attention_norm.weight                  -> blk.39.attn_norm.weight                  | BF16   | [5120]\n",
      "layers.39.ffn_norm.weight                        -> blk.39.ffn_norm.weight                   | BF16   | [5120]\n",
      "skipping tensor rope_freqs\n",
      "Writing models/13B-v2/ggml-model-f16.gguf, format 1\n",
      "gguf: This GGUF file is for Little Endian only\n",
      "gguf: Adding 61249 merge(s).\n",
      "gguf: Setting special token type bos to 1\n",
      "gguf: Setting special token type eos to 2\n",
      "gguf: Setting special token type unk to 0\n",
      "gguf: Setting add_bos_token to True\n",
      "gguf: Setting add_eos_token to False\n",
      "[  1/363] Writing tensor token_embd.weight                      | size  32000 x   5120  | type F16  | T+   1\n",
      "[  2/363] Writing tensor output_norm.weight                     | size   5120           | type F32  | T+   1\n",
      "[  3/363] Writing tensor output.weight                          | size  32000 x   5120  | type F16  | T+   2\n",
      "[  4/363] Writing tensor blk.0.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[  5/363] Writing tensor blk.0.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[  6/363] Writing tensor blk.0.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[  7/363] Writing tensor blk.0.attn_output.weight               | size   5120 x   5120  | type F16  | T+   2\n",
      "[  8/363] Writing tensor blk.0.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   2\n",
      "[  9/363] Writing tensor blk.0.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   2\n",
      "[ 10/363] Writing tensor blk.0.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   2\n",
      "[ 11/363] Writing tensor blk.0.attn_norm.weight                 | size   5120           | type F32  | T+   2\n",
      "[ 12/363] Writing tensor blk.0.ffn_norm.weight                  | size   5120           | type F32  | T+   2\n",
      "[ 13/363] Writing tensor blk.1.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 14/363] Writing tensor blk.1.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 15/363] Writing tensor blk.1.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 16/363] Writing tensor blk.1.attn_output.weight               | size   5120 x   5120  | type F16  | T+   2\n",
      "[ 17/363] Writing tensor blk.1.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   3\n",
      "[ 18/363] Writing tensor blk.1.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   3\n",
      "[ 19/363] Writing tensor blk.1.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   4\n",
      "[ 20/363] Writing tensor blk.1.attn_norm.weight                 | size   5120           | type F32  | T+   4\n",
      "[ 21/363] Writing tensor blk.1.ffn_norm.weight                  | size   5120           | type F32  | T+   4\n",
      "[ 22/363] Writing tensor blk.2.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   4\n",
      "[ 23/363] Writing tensor blk.2.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   4\n",
      "[ 24/363] Writing tensor blk.2.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   4\n",
      "[ 25/363] Writing tensor blk.2.attn_output.weight               | size   5120 x   5120  | type F16  | T+   4\n",
      "[ 26/363] Writing tensor blk.2.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   4\n",
      "[ 27/363] Writing tensor blk.2.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   5\n",
      "[ 28/363] Writing tensor blk.2.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   5\n",
      "[ 29/363] Writing tensor blk.2.attn_norm.weight                 | size   5120           | type F32  | T+   5\n",
      "[ 30/363] Writing tensor blk.2.ffn_norm.weight                  | size   5120           | type F32  | T+   5\n",
      "[ 31/363] Writing tensor blk.3.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 32/363] Writing tensor blk.3.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 33/363] Writing tensor blk.3.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 34/363] Writing tensor blk.3.attn_output.weight               | size   5120 x   5120  | type F16  | T+   5\n",
      "[ 35/363] Writing tensor blk.3.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   6\n",
      "[ 36/363] Writing tensor blk.3.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   6\n",
      "[ 37/363] Writing tensor blk.3.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   6\n",
      "[ 38/363] Writing tensor blk.3.attn_norm.weight                 | size   5120           | type F32  | T+   7\n",
      "[ 39/363] Writing tensor blk.3.ffn_norm.weight                  | size   5120           | type F32  | T+   7\n",
      "[ 40/363] Writing tensor blk.4.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 41/363] Writing tensor blk.4.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 42/363] Writing tensor blk.4.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 43/363] Writing tensor blk.4.attn_output.weight               | size   5120 x   5120  | type F16  | T+   7\n",
      "[ 44/363] Writing tensor blk.4.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   7\n",
      "[ 45/363] Writing tensor blk.4.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   8\n",
      "[ 46/363] Writing tensor blk.4.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   8\n",
      "[ 47/363] Writing tensor blk.4.attn_norm.weight                 | size   5120           | type F32  | T+   8\n",
      "[ 48/363] Writing tensor blk.4.ffn_norm.weight                  | size   5120           | type F32  | T+   8\n",
      "[ 49/363] Writing tensor blk.5.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 50/363] Writing tensor blk.5.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 51/363] Writing tensor blk.5.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 52/363] Writing tensor blk.5.attn_output.weight               | size   5120 x   5120  | type F16  | T+   8\n",
      "[ 53/363] Writing tensor blk.5.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+   8\n",
      "[ 54/363] Writing tensor blk.5.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+   9\n",
      "[ 55/363] Writing tensor blk.5.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+   9\n",
      "[ 56/363] Writing tensor blk.5.attn_norm.weight                 | size   5120           | type F32  | T+   9\n",
      "[ 57/363] Writing tensor blk.5.ffn_norm.weight                  | size   5120           | type F32  | T+   9\n",
      "[ 58/363] Writing tensor blk.6.attn_q.weight                    | size   5120 x   5120  | type F16  | T+   9\n",
      "[ 59/363] Writing tensor blk.6.attn_k.weight                    | size   5120 x   5120  | type F16  | T+   9\n",
      "[ 60/363] Writing tensor blk.6.attn_v.weight                    | size   5120 x   5120  | type F16  | T+   9\n",
      "[ 61/363] Writing tensor blk.6.attn_output.weight               | size   5120 x   5120  | type F16  | T+   9\n",
      "[ 62/363] Writing tensor blk.6.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  10\n",
      "[ 63/363] Writing tensor blk.6.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  10\n",
      "[ 64/363] Writing tensor blk.6.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  11\n",
      "[ 65/363] Writing tensor blk.6.attn_norm.weight                 | size   5120           | type F32  | T+  11\n",
      "[ 66/363] Writing tensor blk.6.ffn_norm.weight                  | size   5120           | type F32  | T+  11\n",
      "[ 67/363] Writing tensor blk.7.attn_q.weight                    | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 68/363] Writing tensor blk.7.attn_k.weight                    | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 69/363] Writing tensor blk.7.attn_v.weight                    | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 70/363] Writing tensor blk.7.attn_output.weight               | size   5120 x   5120  | type F16  | T+  11\n",
      "[ 71/363] Writing tensor blk.7.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  11\n",
      "[ 72/363] Writing tensor blk.7.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  12\n",
      "[ 73/363] Writing tensor blk.7.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  12\n",
      "[ 74/363] Writing tensor blk.7.attn_norm.weight                 | size   5120           | type F32  | T+  12\n",
      "[ 75/363] Writing tensor blk.7.ffn_norm.weight                  | size   5120           | type F32  | T+  12\n",
      "[ 76/363] Writing tensor blk.8.attn_q.weight                    | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 77/363] Writing tensor blk.8.attn_k.weight                    | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 78/363] Writing tensor blk.8.attn_v.weight                    | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 79/363] Writing tensor blk.8.attn_output.weight               | size   5120 x   5120  | type F16  | T+  12\n",
      "[ 80/363] Writing tensor blk.8.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  13\n",
      "[ 81/363] Writing tensor blk.8.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  13\n",
      "[ 82/363] Writing tensor blk.8.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  13\n",
      "[ 83/363] Writing tensor blk.8.attn_norm.weight                 | size   5120           | type F32  | T+  13\n",
      "[ 84/363] Writing tensor blk.8.ffn_norm.weight                  | size   5120           | type F32  | T+  13\n",
      "[ 85/363] Writing tensor blk.9.attn_q.weight                    | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 86/363] Writing tensor blk.9.attn_k.weight                    | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 87/363] Writing tensor blk.9.attn_v.weight                    | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 88/363] Writing tensor blk.9.attn_output.weight               | size   5120 x   5120  | type F16  | T+  13\n",
      "[ 89/363] Writing tensor blk.9.ffn_gate.weight                  | size  13824 x   5120  | type F16  | T+  14\n",
      "[ 90/363] Writing tensor blk.9.ffn_down.weight                  | size   5120 x  13824  | type F16  | T+  14\n",
      "[ 91/363] Writing tensor blk.9.ffn_up.weight                    | size  13824 x   5120  | type F16  | T+  15\n",
      "[ 92/363] Writing tensor blk.9.attn_norm.weight                 | size   5120           | type F32  | T+  15\n",
      "[ 93/363] Writing tensor blk.9.ffn_norm.weight                  | size   5120           | type F32  | T+  15\n",
      "[ 94/363] Writing tensor blk.10.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  15\n",
      "[ 95/363] Writing tensor blk.10.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  15\n",
      "[ 96/363] Writing tensor blk.10.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  15\n",
      "[ 97/363] Writing tensor blk.10.attn_output.weight              | size   5120 x   5120  | type F16  | T+  15\n",
      "[ 98/363] Writing tensor blk.10.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  15\n",
      "[ 99/363] Writing tensor blk.10.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  16\n",
      "[100/363] Writing tensor blk.10.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  16\n",
      "[101/363] Writing tensor blk.10.attn_norm.weight                | size   5120           | type F32  | T+  16\n",
      "[102/363] Writing tensor blk.10.ffn_norm.weight                 | size   5120           | type F32  | T+  16\n",
      "[103/363] Writing tensor blk.11.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  16\n",
      "[104/363] Writing tensor blk.11.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  16\n",
      "[105/363] Writing tensor blk.11.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  16\n",
      "[106/363] Writing tensor blk.11.attn_output.weight              | size   5120 x   5120  | type F16  | T+  16\n",
      "[107/363] Writing tensor blk.11.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  17\n",
      "[108/363] Writing tensor blk.11.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  17\n",
      "[109/363] Writing tensor blk.11.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  17\n",
      "[110/363] Writing tensor blk.11.attn_norm.weight                | size   5120           | type F32  | T+  17\n",
      "[111/363] Writing tensor blk.11.ffn_norm.weight                 | size   5120           | type F32  | T+  17\n",
      "[112/363] Writing tensor blk.12.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  17\n",
      "[113/363] Writing tensor blk.12.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  18\n",
      "[114/363] Writing tensor blk.12.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  18\n",
      "[115/363] Writing tensor blk.12.attn_output.weight              | size   5120 x   5120  | type F16  | T+  18\n",
      "[116/363] Writing tensor blk.12.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  18\n",
      "[117/363] Writing tensor blk.12.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  18\n",
      "[118/363] Writing tensor blk.12.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  19\n",
      "[119/363] Writing tensor blk.12.attn_norm.weight                | size   5120           | type F32  | T+  19\n",
      "[120/363] Writing tensor blk.12.ffn_norm.weight                 | size   5120           | type F32  | T+  19\n",
      "[121/363] Writing tensor blk.13.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  19\n",
      "[122/363] Writing tensor blk.13.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  19\n",
      "[123/363] Writing tensor blk.13.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  19\n",
      "[124/363] Writing tensor blk.13.attn_output.weight              | size   5120 x   5120  | type F16  | T+  19\n",
      "[125/363] Writing tensor blk.13.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  19\n",
      "[126/363] Writing tensor blk.13.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  20\n",
      "[127/363] Writing tensor blk.13.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  20\n",
      "[128/363] Writing tensor blk.13.attn_norm.weight                | size   5120           | type F32  | T+  20\n",
      "[129/363] Writing tensor blk.13.ffn_norm.weight                 | size   5120           | type F32  | T+  20\n",
      "[130/363] Writing tensor blk.14.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  20\n",
      "[131/363] Writing tensor blk.14.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  20\n",
      "[132/363] Writing tensor blk.14.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  20\n",
      "[133/363] Writing tensor blk.14.attn_output.weight              | size   5120 x   5120  | type F16  | T+  20\n",
      "[134/363] Writing tensor blk.14.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  21\n",
      "[135/363] Writing tensor blk.14.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  21\n",
      "[136/363] Writing tensor blk.14.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  22\n",
      "[137/363] Writing tensor blk.14.attn_norm.weight                | size   5120           | type F32  | T+  22\n",
      "[138/363] Writing tensor blk.14.ffn_norm.weight                 | size   5120           | type F32  | T+  22\n",
      "[139/363] Writing tensor blk.15.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  22\n",
      "[140/363] Writing tensor blk.15.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  22\n",
      "[141/363] Writing tensor blk.15.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  22\n",
      "[142/363] Writing tensor blk.15.attn_output.weight              | size   5120 x   5120  | type F16  | T+  22\n",
      "[143/363] Writing tensor blk.15.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  22\n",
      "[144/363] Writing tensor blk.15.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  23\n",
      "[145/363] Writing tensor blk.15.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  23\n",
      "[146/363] Writing tensor blk.15.attn_norm.weight                | size   5120           | type F32  | T+  23\n",
      "[147/363] Writing tensor blk.15.ffn_norm.weight                 | size   5120           | type F32  | T+  23\n",
      "[148/363] Writing tensor blk.16.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  23\n",
      "[149/363] Writing tensor blk.16.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  23\n",
      "[150/363] Writing tensor blk.16.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  23\n",
      "[151/363] Writing tensor blk.16.attn_output.weight              | size   5120 x   5120  | type F16  | T+  23\n",
      "[152/363] Writing tensor blk.16.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  24\n",
      "[153/363] Writing tensor blk.16.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  24\n",
      "[154/363] Writing tensor blk.16.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  24\n",
      "[155/363] Writing tensor blk.16.attn_norm.weight                | size   5120           | type F32  | T+  24\n",
      "[156/363] Writing tensor blk.16.ffn_norm.weight                 | size   5120           | type F32  | T+  24\n",
      "[157/363] Writing tensor blk.17.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  24\n",
      "[158/363] Writing tensor blk.17.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  25\n",
      "[159/363] Writing tensor blk.17.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  25\n",
      "[160/363] Writing tensor blk.17.attn_output.weight              | size   5120 x   5120  | type F16  | T+  25\n",
      "[161/363] Writing tensor blk.17.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  25\n",
      "[162/363] Writing tensor blk.17.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  25\n",
      "[163/363] Writing tensor blk.17.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  26\n",
      "[164/363] Writing tensor blk.17.attn_norm.weight                | size   5120           | type F32  | T+  26\n",
      "[165/363] Writing tensor blk.17.ffn_norm.weight                 | size   5120           | type F32  | T+  26\n",
      "[166/363] Writing tensor blk.18.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  26\n",
      "[167/363] Writing tensor blk.18.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  26\n",
      "[168/363] Writing tensor blk.18.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  26\n",
      "[169/363] Writing tensor blk.18.attn_output.weight              | size   5120 x   5120  | type F16  | T+  26\n",
      "[170/363] Writing tensor blk.18.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  27\n",
      "[171/363] Writing tensor blk.18.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  27\n",
      "[172/363] Writing tensor blk.18.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  27\n",
      "[173/363] Writing tensor blk.18.attn_norm.weight                | size   5120           | type F32  | T+  27\n",
      "[174/363] Writing tensor blk.18.ffn_norm.weight                 | size   5120           | type F32  | T+  27\n",
      "[175/363] Writing tensor blk.19.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  27\n",
      "[176/363] Writing tensor blk.19.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  27\n",
      "[177/363] Writing tensor blk.19.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  27\n",
      "[178/363] Writing tensor blk.19.attn_output.weight              | size   5120 x   5120  | type F16  | T+  27\n",
      "[179/363] Writing tensor blk.19.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  28\n",
      "[180/363] Writing tensor blk.19.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  28\n",
      "[181/363] Writing tensor blk.19.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  29\n",
      "[182/363] Writing tensor blk.19.attn_norm.weight                | size   5120           | type F32  | T+  29\n",
      "[183/363] Writing tensor blk.19.ffn_norm.weight                 | size   5120           | type F32  | T+  29\n",
      "[184/363] Writing tensor blk.20.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  29\n",
      "[185/363] Writing tensor blk.20.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  29\n",
      "[186/363] Writing tensor blk.20.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  29\n",
      "[187/363] Writing tensor blk.20.attn_output.weight              | size   5120 x   5120  | type F16  | T+  29\n",
      "[188/363] Writing tensor blk.20.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  29\n",
      "[189/363] Writing tensor blk.20.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  30\n",
      "[190/363] Writing tensor blk.20.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  30\n",
      "[191/363] Writing tensor blk.20.attn_norm.weight                | size   5120           | type F32  | T+  30\n",
      "[192/363] Writing tensor blk.20.ffn_norm.weight                 | size   5120           | type F32  | T+  30\n",
      "[193/363] Writing tensor blk.21.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  30\n",
      "[194/363] Writing tensor blk.21.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  30\n",
      "[195/363] Writing tensor blk.21.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  30\n",
      "[196/363] Writing tensor blk.21.attn_output.weight              | size   5120 x   5120  | type F16  | T+  30\n",
      "[197/363] Writing tensor blk.21.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  31\n",
      "[198/363] Writing tensor blk.21.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  31\n",
      "[199/363] Writing tensor blk.21.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  31\n",
      "[200/363] Writing tensor blk.21.attn_norm.weight                | size   5120           | type F32  | T+  31\n",
      "[201/363] Writing tensor blk.21.ffn_norm.weight                 | size   5120           | type F32  | T+  31\n",
      "[202/363] Writing tensor blk.22.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  31\n",
      "[203/363] Writing tensor blk.22.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  32\n",
      "[204/363] Writing tensor blk.22.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  32\n",
      "[205/363] Writing tensor blk.22.attn_output.weight              | size   5120 x   5120  | type F16  | T+  32\n",
      "[206/363] Writing tensor blk.22.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  33\n",
      "[207/363] Writing tensor blk.22.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  34\n",
      "[208/363] Writing tensor blk.22.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  35\n",
      "[209/363] Writing tensor blk.22.attn_norm.weight                | size   5120           | type F32  | T+  35\n",
      "[210/363] Writing tensor blk.22.ffn_norm.weight                 | size   5120           | type F32  | T+  35\n",
      "[211/363] Writing tensor blk.23.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  35\n",
      "[212/363] Writing tensor blk.23.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  35\n",
      "[213/363] Writing tensor blk.23.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  35\n",
      "[214/363] Writing tensor blk.23.attn_output.weight              | size   5120 x   5120  | type F16  | T+  35\n",
      "[215/363] Writing tensor blk.23.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  35\n",
      "[216/363] Writing tensor blk.23.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  36\n",
      "[217/363] Writing tensor blk.23.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  37\n",
      "[218/363] Writing tensor blk.23.attn_norm.weight                | size   5120           | type F32  | T+  37\n",
      "[219/363] Writing tensor blk.23.ffn_norm.weight                 | size   5120           | type F32  | T+  37\n",
      "[220/363] Writing tensor blk.24.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  37\n",
      "[221/363] Writing tensor blk.24.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  37\n",
      "[222/363] Writing tensor blk.24.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  37\n",
      "[223/363] Writing tensor blk.24.attn_output.weight              | size   5120 x   5120  | type F16  | T+  37\n",
      "[224/363] Writing tensor blk.24.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  37\n",
      "[225/363] Writing tensor blk.24.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  38\n",
      "[226/363] Writing tensor blk.24.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  38\n",
      "[227/363] Writing tensor blk.24.attn_norm.weight                | size   5120           | type F32  | T+  38\n",
      "[228/363] Writing tensor blk.24.ffn_norm.weight                 | size   5120           | type F32  | T+  38\n",
      "[229/363] Writing tensor blk.25.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  38\n",
      "[230/363] Writing tensor blk.25.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  38\n",
      "[231/363] Writing tensor blk.25.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  38\n",
      "[232/363] Writing tensor blk.25.attn_output.weight              | size   5120 x   5120  | type F16  | T+  38\n",
      "[233/363] Writing tensor blk.25.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  39\n",
      "[234/363] Writing tensor blk.25.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  39\n",
      "[235/363] Writing tensor blk.25.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  39\n",
      "[236/363] Writing tensor blk.25.attn_norm.weight                | size   5120           | type F32  | T+  40\n",
      "[237/363] Writing tensor blk.25.ffn_norm.weight                 | size   5120           | type F32  | T+  40\n",
      "[238/363] Writing tensor blk.26.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  40\n",
      "[239/363] Writing tensor blk.26.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  40\n",
      "[240/363] Writing tensor blk.26.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  40\n",
      "[241/363] Writing tensor blk.26.attn_output.weight              | size   5120 x   5120  | type F16  | T+  40\n",
      "[242/363] Writing tensor blk.26.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  40\n",
      "[243/363] Writing tensor blk.26.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  41\n",
      "[244/363] Writing tensor blk.26.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  41\n",
      "[245/363] Writing tensor blk.26.attn_norm.weight                | size   5120           | type F32  | T+  41\n",
      "[246/363] Writing tensor blk.26.ffn_norm.weight                 | size   5120           | type F32  | T+  41\n",
      "[247/363] Writing tensor blk.27.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  41\n",
      "[248/363] Writing tensor blk.27.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  41\n",
      "[249/363] Writing tensor blk.27.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  41\n",
      "[250/363] Writing tensor blk.27.attn_output.weight              | size   5120 x   5120  | type F16  | T+  41\n",
      "[251/363] Writing tensor blk.27.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  41\n",
      "[252/363] Writing tensor blk.27.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  42\n",
      "[253/363] Writing tensor blk.27.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  42\n",
      "[254/363] Writing tensor blk.27.attn_norm.weight                | size   5120           | type F32  | T+  43\n",
      "[255/363] Writing tensor blk.27.ffn_norm.weight                 | size   5120           | type F32  | T+  43\n",
      "[256/363] Writing tensor blk.28.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  43\n",
      "[257/363] Writing tensor blk.28.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  43\n",
      "[258/363] Writing tensor blk.28.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  43\n",
      "[259/363] Writing tensor blk.28.attn_output.weight              | size   5120 x   5120  | type F16  | T+  43\n",
      "[260/363] Writing tensor blk.28.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  43\n",
      "[261/363] Writing tensor blk.28.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  44\n",
      "[262/363] Writing tensor blk.28.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  44\n",
      "[263/363] Writing tensor blk.28.attn_norm.weight                | size   5120           | type F32  | T+  44\n",
      "[264/363] Writing tensor blk.28.ffn_norm.weight                 | size   5120           | type F32  | T+  44\n",
      "[265/363] Writing tensor blk.29.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  44\n",
      "[266/363] Writing tensor blk.29.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  44\n",
      "[267/363] Writing tensor blk.29.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  44\n",
      "[268/363] Writing tensor blk.29.attn_output.weight              | size   5120 x   5120  | type F16  | T+  44\n",
      "[269/363] Writing tensor blk.29.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  44\n",
      "[270/363] Writing tensor blk.29.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  45\n",
      "[271/363] Writing tensor blk.29.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  45\n",
      "[272/363] Writing tensor blk.29.attn_norm.weight                | size   5120           | type F32  | T+  45\n",
      "[273/363] Writing tensor blk.29.ffn_norm.weight                 | size   5120           | type F32  | T+  45\n",
      "[274/363] Writing tensor blk.30.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  45\n",
      "[275/363] Writing tensor blk.30.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  45\n",
      "[276/363] Writing tensor blk.30.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  45\n",
      "[277/363] Writing tensor blk.30.attn_output.weight              | size   5120 x   5120  | type F16  | T+  45\n",
      "[278/363] Writing tensor blk.30.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  46\n",
      "[279/363] Writing tensor blk.30.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  46\n",
      "[280/363] Writing tensor blk.30.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  47\n",
      "[281/363] Writing tensor blk.30.attn_norm.weight                | size   5120           | type F32  | T+  47\n",
      "[282/363] Writing tensor blk.30.ffn_norm.weight                 | size   5120           | type F32  | T+  47\n",
      "[283/363] Writing tensor blk.31.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  47\n",
      "[284/363] Writing tensor blk.31.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  47\n",
      "[285/363] Writing tensor blk.31.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  47\n",
      "[286/363] Writing tensor blk.31.attn_output.weight              | size   5120 x   5120  | type F16  | T+  47\n",
      "[287/363] Writing tensor blk.31.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  47\n",
      "[288/363] Writing tensor blk.31.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  48\n",
      "[289/363] Writing tensor blk.31.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  48\n",
      "[290/363] Writing tensor blk.31.attn_norm.weight                | size   5120           | type F32  | T+  48\n",
      "[291/363] Writing tensor blk.31.ffn_norm.weight                 | size   5120           | type F32  | T+  48\n",
      "[292/363] Writing tensor blk.32.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  48\n",
      "[293/363] Writing tensor blk.32.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  48\n",
      "[294/363] Writing tensor blk.32.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  48\n",
      "[295/363] Writing tensor blk.32.attn_output.weight              | size   5120 x   5120  | type F16  | T+  48\n",
      "[296/363] Writing tensor blk.32.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  49\n",
      "[297/363] Writing tensor blk.32.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  49\n",
      "[298/363] Writing tensor blk.32.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  50\n",
      "[299/363] Writing tensor blk.32.attn_norm.weight                | size   5120           | type F32  | T+  50\n",
      "[300/363] Writing tensor blk.32.ffn_norm.weight                 | size   5120           | type F32  | T+  50\n",
      "[301/363] Writing tensor blk.33.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  50\n",
      "[302/363] Writing tensor blk.33.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  50\n",
      "[303/363] Writing tensor blk.33.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  50\n",
      "[304/363] Writing tensor blk.33.attn_output.weight              | size   5120 x   5120  | type F16  | T+  50\n",
      "[305/363] Writing tensor blk.33.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  50\n",
      "[306/363] Writing tensor blk.33.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  51\n",
      "[307/363] Writing tensor blk.33.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  51\n",
      "[308/363] Writing tensor blk.33.attn_norm.weight                | size   5120           | type F32  | T+  51\n",
      "[309/363] Writing tensor blk.33.ffn_norm.weight                 | size   5120           | type F32  | T+  51\n",
      "[310/363] Writing tensor blk.34.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  51\n",
      "[311/363] Writing tensor blk.34.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  51\n",
      "[312/363] Writing tensor blk.34.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  51\n",
      "[313/363] Writing tensor blk.34.attn_output.weight              | size   5120 x   5120  | type F16  | T+  51\n",
      "[314/363] Writing tensor blk.34.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  52\n",
      "[315/363] Writing tensor blk.34.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  52\n",
      "[316/363] Writing tensor blk.34.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  52\n",
      "[317/363] Writing tensor blk.34.attn_norm.weight                | size   5120           | type F32  | T+  53\n",
      "[318/363] Writing tensor blk.34.ffn_norm.weight                 | size   5120           | type F32  | T+  53\n",
      "[319/363] Writing tensor blk.35.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  53\n",
      "[320/363] Writing tensor blk.35.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  53\n",
      "[321/363] Writing tensor blk.35.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  53\n",
      "[322/363] Writing tensor blk.35.attn_output.weight              | size   5120 x   5120  | type F16  | T+  53\n",
      "[323/363] Writing tensor blk.35.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  53\n",
      "[324/363] Writing tensor blk.35.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  54\n",
      "[325/363] Writing tensor blk.35.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  54\n",
      "[326/363] Writing tensor blk.35.attn_norm.weight                | size   5120           | type F32  | T+  54\n",
      "[327/363] Writing tensor blk.35.ffn_norm.weight                 | size   5120           | type F32  | T+  54\n",
      "[328/363] Writing tensor blk.36.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  54\n",
      "[329/363] Writing tensor blk.36.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  54\n",
      "[330/363] Writing tensor blk.36.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  54\n",
      "[331/363] Writing tensor blk.36.attn_output.weight              | size   5120 x   5120  | type F16  | T+  54\n",
      "[332/363] Writing tensor blk.36.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  54\n",
      "[333/363] Writing tensor blk.36.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  55\n",
      "[334/363] Writing tensor blk.36.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  55\n",
      "[335/363] Writing tensor blk.36.attn_norm.weight                | size   5120           | type F32  | T+  55\n",
      "[336/363] Writing tensor blk.36.ffn_norm.weight                 | size   5120           | type F32  | T+  55\n",
      "[337/363] Writing tensor blk.37.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  55\n",
      "[338/363] Writing tensor blk.37.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  55\n",
      "[339/363] Writing tensor blk.37.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  55\n",
      "[340/363] Writing tensor blk.37.attn_output.weight              | size   5120 x   5120  | type F16  | T+  55\n",
      "[341/363] Writing tensor blk.37.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  56\n",
      "[342/363] Writing tensor blk.37.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  56\n",
      "[343/363] Writing tensor blk.37.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  57\n",
      "[344/363] Writing tensor blk.37.attn_norm.weight                | size   5120           | type F32  | T+  57\n",
      "[345/363] Writing tensor blk.37.ffn_norm.weight                 | size   5120           | type F32  | T+  57\n",
      "[346/363] Writing tensor blk.38.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  57\n",
      "[347/363] Writing tensor blk.38.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  57\n",
      "[348/363] Writing tensor blk.38.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  57\n",
      "[349/363] Writing tensor blk.38.attn_output.weight              | size   5120 x   5120  | type F16  | T+  57\n",
      "[350/363] Writing tensor blk.38.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  57\n",
      "[351/363] Writing tensor blk.38.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  58\n",
      "[352/363] Writing tensor blk.38.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  58\n",
      "[353/363] Writing tensor blk.38.attn_norm.weight                | size   5120           | type F32  | T+  58\n",
      "[354/363] Writing tensor blk.38.ffn_norm.weight                 | size   5120           | type F32  | T+  58\n",
      "[355/363] Writing tensor blk.39.attn_q.weight                   | size   5120 x   5120  | type F16  | T+  58\n",
      "[356/363] Writing tensor blk.39.attn_k.weight                   | size   5120 x   5120  | type F16  | T+  58\n",
      "[357/363] Writing tensor blk.39.attn_v.weight                   | size   5120 x   5120  | type F16  | T+  58\n",
      "[358/363] Writing tensor blk.39.attn_output.weight              | size   5120 x   5120  | type F16  | T+  58\n",
      "[359/363] Writing tensor blk.39.ffn_gate.weight                 | size  13824 x   5120  | type F16  | T+  59\n",
      "[360/363] Writing tensor blk.39.ffn_down.weight                 | size   5120 x  13824  | type F16  | T+  59\n",
      "[361/363] Writing tensor blk.39.ffn_up.weight                   | size  13824 x   5120  | type F16  | T+  59\n",
      "[362/363] Writing tensor blk.39.attn_norm.weight                | size   5120           | type F32  | T+  59\n",
      "[363/363] Writing tensor blk.39.ffn_norm.weight                 | size   5120           | type F32  | T+  59\n",
      "Wrote models/13B-v2/ggml-model-f16.gguf\n",
      "Loading model file models/70B-v2/consolidated.00.pth\n",
      "Loading model file models/70B-v2/consolidated.01.pth\n",
      "Loading model file models/70B-v2/consolidated.02.pth\n",
      "Loading model file models/70B-v2/consolidated.03.pth\n",
      "Loading model file models/70B-v2/consolidated.04.pth\n",
      "Loading model file models/70B-v2/consolidated.05.pth\n",
      "Loading model file models/70B-v2/consolidated.06.pth\n",
      "Loading model file models/70B-v2/consolidated.07.pth\n",
      "params = Params(n_vocab=32000, n_embd=8192, n_layer=80, n_ctx=4096, n_ff=28672, n_head=64, n_head_kv=8, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=None, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('models/70B-v2'))\n",
      "32000 32000\n",
      "Vocab info: <VocabLoader with 32000 base tokens and 0 added tokens>\n",
      "Special vocab info: <SpecialVocab with 61249 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0}, add special tokens {'bos': True, 'eos': False}>\n",
      "tok_embeddings.weight                            -> token_embd.weight                        | BF16   | [32000, 8192]\n",
      "norm.weight                                      -> output_norm.weight                       | BF16   | [8192]\n",
      "output.weight                                    -> output.weight                            | BF16   | [32000, 8192]\n",
      "layers.0.attention.wq.weight                     -> blk.0.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.0.attention.wk.weight                     -> blk.0.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.0.attention.wv.weight                     -> blk.0.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.0.attention.wo.weight                     -> blk.0.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.0.feed_forward.w1.weight                  -> blk.0.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.0.feed_forward.w2.weight                  -> blk.0.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.0.feed_forward.w3.weight                  -> blk.0.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.0.attention_norm.weight                   -> blk.0.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.0.ffn_norm.weight                         -> blk.0.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.1.attention.wq.weight                     -> blk.1.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.1.attention.wk.weight                     -> blk.1.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.1.attention.wv.weight                     -> blk.1.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.1.attention.wo.weight                     -> blk.1.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.1.feed_forward.w1.weight                  -> blk.1.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.1.feed_forward.w2.weight                  -> blk.1.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.1.feed_forward.w3.weight                  -> blk.1.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.1.attention_norm.weight                   -> blk.1.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.1.ffn_norm.weight                         -> blk.1.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.2.attention.wq.weight                     -> blk.2.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.2.attention.wk.weight                     -> blk.2.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.2.attention.wv.weight                     -> blk.2.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.2.attention.wo.weight                     -> blk.2.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.2.feed_forward.w1.weight                  -> blk.2.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.2.feed_forward.w2.weight                  -> blk.2.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.2.feed_forward.w3.weight                  -> blk.2.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.2.attention_norm.weight                   -> blk.2.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.2.ffn_norm.weight                         -> blk.2.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.3.attention.wq.weight                     -> blk.3.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.3.attention.wk.weight                     -> blk.3.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.3.attention.wv.weight                     -> blk.3.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.3.attention.wo.weight                     -> blk.3.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.3.feed_forward.w1.weight                  -> blk.3.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.3.feed_forward.w2.weight                  -> blk.3.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.3.feed_forward.w3.weight                  -> blk.3.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.3.attention_norm.weight                   -> blk.3.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.3.ffn_norm.weight                         -> blk.3.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.4.attention.wq.weight                     -> blk.4.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.4.attention.wk.weight                     -> blk.4.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.4.attention.wv.weight                     -> blk.4.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.4.attention.wo.weight                     -> blk.4.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.4.feed_forward.w1.weight                  -> blk.4.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.4.feed_forward.w2.weight                  -> blk.4.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.4.feed_forward.w3.weight                  -> blk.4.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.4.attention_norm.weight                   -> blk.4.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.4.ffn_norm.weight                         -> blk.4.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.5.attention.wq.weight                     -> blk.5.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.5.attention.wk.weight                     -> blk.5.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.5.attention.wv.weight                     -> blk.5.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.5.attention.wo.weight                     -> blk.5.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.5.feed_forward.w1.weight                  -> blk.5.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.5.feed_forward.w2.weight                  -> blk.5.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.5.feed_forward.w3.weight                  -> blk.5.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.5.attention_norm.weight                   -> blk.5.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.5.ffn_norm.weight                         -> blk.5.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.6.attention.wq.weight                     -> blk.6.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.6.attention.wk.weight                     -> blk.6.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.6.attention.wv.weight                     -> blk.6.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.6.attention.wo.weight                     -> blk.6.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.6.feed_forward.w1.weight                  -> blk.6.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.6.feed_forward.w2.weight                  -> blk.6.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.6.feed_forward.w3.weight                  -> blk.6.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.6.attention_norm.weight                   -> blk.6.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.6.ffn_norm.weight                         -> blk.6.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.7.attention.wq.weight                     -> blk.7.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.7.attention.wk.weight                     -> blk.7.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.7.attention.wv.weight                     -> blk.7.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.7.attention.wo.weight                     -> blk.7.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.7.feed_forward.w1.weight                  -> blk.7.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.7.feed_forward.w2.weight                  -> blk.7.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.7.feed_forward.w3.weight                  -> blk.7.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.7.attention_norm.weight                   -> blk.7.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.7.ffn_norm.weight                         -> blk.7.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.8.attention.wq.weight                     -> blk.8.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.8.attention.wk.weight                     -> blk.8.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.8.attention.wv.weight                     -> blk.8.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.8.attention.wo.weight                     -> blk.8.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.8.feed_forward.w1.weight                  -> blk.8.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.8.feed_forward.w2.weight                  -> blk.8.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.8.feed_forward.w3.weight                  -> blk.8.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.8.attention_norm.weight                   -> blk.8.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.8.ffn_norm.weight                         -> blk.8.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.9.attention.wq.weight                     -> blk.9.attn_q.weight                      | BF16   | [8192, 8192]\n",
      "layers.9.attention.wk.weight                     -> blk.9.attn_k.weight                      | BF16   | [1024, 8192]\n",
      "layers.9.attention.wv.weight                     -> blk.9.attn_v.weight                      | BF16   | [1024, 8192]\n",
      "layers.9.attention.wo.weight                     -> blk.9.attn_output.weight                 | BF16   | [8192, 8192]\n",
      "layers.9.feed_forward.w1.weight                  -> blk.9.ffn_gate.weight                    | BF16   | [28672, 8192]\n",
      "layers.9.feed_forward.w2.weight                  -> blk.9.ffn_down.weight                    | BF16   | [8192, 28672]\n",
      "layers.9.feed_forward.w3.weight                  -> blk.9.ffn_up.weight                      | BF16   | [28672, 8192]\n",
      "layers.9.attention_norm.weight                   -> blk.9.attn_norm.weight                   | BF16   | [8192]\n",
      "layers.9.ffn_norm.weight                         -> blk.9.ffn_norm.weight                    | BF16   | [8192]\n",
      "layers.10.attention.wq.weight                    -> blk.10.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.10.attention.wk.weight                    -> blk.10.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.10.attention.wv.weight                    -> blk.10.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.10.attention.wo.weight                    -> blk.10.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.10.feed_forward.w1.weight                 -> blk.10.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.10.feed_forward.w2.weight                 -> blk.10.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.10.feed_forward.w3.weight                 -> blk.10.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.10.attention_norm.weight                  -> blk.10.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.10.ffn_norm.weight                        -> blk.10.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.11.attention.wq.weight                    -> blk.11.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.11.attention.wk.weight                    -> blk.11.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.11.attention.wv.weight                    -> blk.11.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.11.attention.wo.weight                    -> blk.11.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.11.feed_forward.w1.weight                 -> blk.11.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.11.feed_forward.w2.weight                 -> blk.11.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.11.feed_forward.w3.weight                 -> blk.11.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.11.attention_norm.weight                  -> blk.11.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.11.ffn_norm.weight                        -> blk.11.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.12.attention.wq.weight                    -> blk.12.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.12.attention.wk.weight                    -> blk.12.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.12.attention.wv.weight                    -> blk.12.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.12.attention.wo.weight                    -> blk.12.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.12.feed_forward.w1.weight                 -> blk.12.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.12.feed_forward.w2.weight                 -> blk.12.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.12.feed_forward.w3.weight                 -> blk.12.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.12.attention_norm.weight                  -> blk.12.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.12.ffn_norm.weight                        -> blk.12.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.13.attention.wq.weight                    -> blk.13.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.13.attention.wk.weight                    -> blk.13.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.13.attention.wv.weight                    -> blk.13.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.13.attention.wo.weight                    -> blk.13.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.13.feed_forward.w1.weight                 -> blk.13.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.13.feed_forward.w2.weight                 -> blk.13.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.13.feed_forward.w3.weight                 -> blk.13.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.13.attention_norm.weight                  -> blk.13.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.13.ffn_norm.weight                        -> blk.13.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.14.attention.wq.weight                    -> blk.14.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.14.attention.wk.weight                    -> blk.14.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.14.attention.wv.weight                    -> blk.14.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.14.attention.wo.weight                    -> blk.14.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.14.feed_forward.w1.weight                 -> blk.14.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.14.feed_forward.w2.weight                 -> blk.14.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.14.feed_forward.w3.weight                 -> blk.14.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.14.attention_norm.weight                  -> blk.14.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.14.ffn_norm.weight                        -> blk.14.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.15.attention.wq.weight                    -> blk.15.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.15.attention.wk.weight                    -> blk.15.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.15.attention.wv.weight                    -> blk.15.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.15.attention.wo.weight                    -> blk.15.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.15.feed_forward.w1.weight                 -> blk.15.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.15.feed_forward.w2.weight                 -> blk.15.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.15.feed_forward.w3.weight                 -> blk.15.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.15.attention_norm.weight                  -> blk.15.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.15.ffn_norm.weight                        -> blk.15.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.16.attention.wq.weight                    -> blk.16.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.16.attention.wk.weight                    -> blk.16.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.16.attention.wv.weight                    -> blk.16.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.16.attention.wo.weight                    -> blk.16.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.16.feed_forward.w1.weight                 -> blk.16.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.16.feed_forward.w2.weight                 -> blk.16.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.16.feed_forward.w3.weight                 -> blk.16.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.16.attention_norm.weight                  -> blk.16.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.16.ffn_norm.weight                        -> blk.16.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.17.attention.wq.weight                    -> blk.17.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.17.attention.wk.weight                    -> blk.17.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.17.attention.wv.weight                    -> blk.17.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.17.attention.wo.weight                    -> blk.17.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.17.feed_forward.w1.weight                 -> blk.17.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.17.feed_forward.w2.weight                 -> blk.17.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.17.feed_forward.w3.weight                 -> blk.17.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.17.attention_norm.weight                  -> blk.17.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.17.ffn_norm.weight                        -> blk.17.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.18.attention.wq.weight                    -> blk.18.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.18.attention.wk.weight                    -> blk.18.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.18.attention.wv.weight                    -> blk.18.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.18.attention.wo.weight                    -> blk.18.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.18.feed_forward.w1.weight                 -> blk.18.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.18.feed_forward.w2.weight                 -> blk.18.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.18.feed_forward.w3.weight                 -> blk.18.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.18.attention_norm.weight                  -> blk.18.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.18.ffn_norm.weight                        -> blk.18.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.19.attention.wq.weight                    -> blk.19.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.19.attention.wk.weight                    -> blk.19.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.19.attention.wv.weight                    -> blk.19.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.19.attention.wo.weight                    -> blk.19.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.19.feed_forward.w1.weight                 -> blk.19.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.19.feed_forward.w2.weight                 -> blk.19.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.19.feed_forward.w3.weight                 -> blk.19.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.19.attention_norm.weight                  -> blk.19.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.19.ffn_norm.weight                        -> blk.19.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.20.attention.wq.weight                    -> blk.20.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.20.attention.wk.weight                    -> blk.20.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.20.attention.wv.weight                    -> blk.20.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.20.attention.wo.weight                    -> blk.20.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.20.feed_forward.w1.weight                 -> blk.20.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.20.feed_forward.w2.weight                 -> blk.20.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.20.feed_forward.w3.weight                 -> blk.20.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.20.attention_norm.weight                  -> blk.20.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.20.ffn_norm.weight                        -> blk.20.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.21.attention.wq.weight                    -> blk.21.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.21.attention.wk.weight                    -> blk.21.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.21.attention.wv.weight                    -> blk.21.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.21.attention.wo.weight                    -> blk.21.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.21.feed_forward.w1.weight                 -> blk.21.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.21.feed_forward.w2.weight                 -> blk.21.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.21.feed_forward.w3.weight                 -> blk.21.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.21.attention_norm.weight                  -> blk.21.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.21.ffn_norm.weight                        -> blk.21.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.22.attention.wq.weight                    -> blk.22.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.22.attention.wk.weight                    -> blk.22.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.22.attention.wv.weight                    -> blk.22.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.22.attention.wo.weight                    -> blk.22.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.22.feed_forward.w1.weight                 -> blk.22.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.22.feed_forward.w2.weight                 -> blk.22.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.22.feed_forward.w3.weight                 -> blk.22.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.22.attention_norm.weight                  -> blk.22.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.22.ffn_norm.weight                        -> blk.22.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.23.attention.wq.weight                    -> blk.23.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.23.attention.wk.weight                    -> blk.23.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.23.attention.wv.weight                    -> blk.23.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.23.attention.wo.weight                    -> blk.23.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.23.feed_forward.w1.weight                 -> blk.23.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.23.feed_forward.w2.weight                 -> blk.23.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.23.feed_forward.w3.weight                 -> blk.23.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.23.attention_norm.weight                  -> blk.23.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.23.ffn_norm.weight                        -> blk.23.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.24.attention.wq.weight                    -> blk.24.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.24.attention.wk.weight                    -> blk.24.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.24.attention.wv.weight                    -> blk.24.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.24.attention.wo.weight                    -> blk.24.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.24.feed_forward.w1.weight                 -> blk.24.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.24.feed_forward.w2.weight                 -> blk.24.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.24.feed_forward.w3.weight                 -> blk.24.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.24.attention_norm.weight                  -> blk.24.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.24.ffn_norm.weight                        -> blk.24.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.25.attention.wq.weight                    -> blk.25.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.25.attention.wk.weight                    -> blk.25.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.25.attention.wv.weight                    -> blk.25.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.25.attention.wo.weight                    -> blk.25.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.25.feed_forward.w1.weight                 -> blk.25.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.25.feed_forward.w2.weight                 -> blk.25.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.25.feed_forward.w3.weight                 -> blk.25.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.25.attention_norm.weight                  -> blk.25.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.25.ffn_norm.weight                        -> blk.25.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.26.attention.wq.weight                    -> blk.26.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.26.attention.wk.weight                    -> blk.26.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.26.attention.wv.weight                    -> blk.26.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.26.attention.wo.weight                    -> blk.26.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.26.feed_forward.w1.weight                 -> blk.26.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.26.feed_forward.w2.weight                 -> blk.26.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.26.feed_forward.w3.weight                 -> blk.26.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.26.attention_norm.weight                  -> blk.26.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.26.ffn_norm.weight                        -> blk.26.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.27.attention.wq.weight                    -> blk.27.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.27.attention.wk.weight                    -> blk.27.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.27.attention.wv.weight                    -> blk.27.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.27.attention.wo.weight                    -> blk.27.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.27.feed_forward.w1.weight                 -> blk.27.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.27.feed_forward.w2.weight                 -> blk.27.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.27.feed_forward.w3.weight                 -> blk.27.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.27.attention_norm.weight                  -> blk.27.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.27.ffn_norm.weight                        -> blk.27.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.28.attention.wq.weight                    -> blk.28.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.28.attention.wk.weight                    -> blk.28.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.28.attention.wv.weight                    -> blk.28.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.28.attention.wo.weight                    -> blk.28.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.28.feed_forward.w1.weight                 -> blk.28.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.28.feed_forward.w2.weight                 -> blk.28.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.28.feed_forward.w3.weight                 -> blk.28.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.28.attention_norm.weight                  -> blk.28.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.28.ffn_norm.weight                        -> blk.28.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.29.attention.wq.weight                    -> blk.29.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.29.attention.wk.weight                    -> blk.29.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.29.attention.wv.weight                    -> blk.29.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.29.attention.wo.weight                    -> blk.29.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.29.feed_forward.w1.weight                 -> blk.29.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.29.feed_forward.w2.weight                 -> blk.29.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.29.feed_forward.w3.weight                 -> blk.29.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.29.attention_norm.weight                  -> blk.29.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.29.ffn_norm.weight                        -> blk.29.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.30.attention.wq.weight                    -> blk.30.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.30.attention.wk.weight                    -> blk.30.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.30.attention.wv.weight                    -> blk.30.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.30.attention.wo.weight                    -> blk.30.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.30.feed_forward.w1.weight                 -> blk.30.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.30.feed_forward.w2.weight                 -> blk.30.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.30.feed_forward.w3.weight                 -> blk.30.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.30.attention_norm.weight                  -> blk.30.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.30.ffn_norm.weight                        -> blk.30.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.31.attention.wq.weight                    -> blk.31.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.31.attention.wk.weight                    -> blk.31.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.31.attention.wv.weight                    -> blk.31.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.31.attention.wo.weight                    -> blk.31.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.31.feed_forward.w1.weight                 -> blk.31.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.31.feed_forward.w2.weight                 -> blk.31.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.31.feed_forward.w3.weight                 -> blk.31.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.31.attention_norm.weight                  -> blk.31.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.31.ffn_norm.weight                        -> blk.31.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.32.attention.wq.weight                    -> blk.32.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.32.attention.wk.weight                    -> blk.32.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.32.attention.wv.weight                    -> blk.32.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.32.attention.wo.weight                    -> blk.32.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.32.feed_forward.w1.weight                 -> blk.32.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.32.feed_forward.w2.weight                 -> blk.32.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.32.feed_forward.w3.weight                 -> blk.32.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.32.attention_norm.weight                  -> blk.32.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.32.ffn_norm.weight                        -> blk.32.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.33.attention.wq.weight                    -> blk.33.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.33.attention.wk.weight                    -> blk.33.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.33.attention.wv.weight                    -> blk.33.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.33.attention.wo.weight                    -> blk.33.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.33.feed_forward.w1.weight                 -> blk.33.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.33.feed_forward.w2.weight                 -> blk.33.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.33.feed_forward.w3.weight                 -> blk.33.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.33.attention_norm.weight                  -> blk.33.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.33.ffn_norm.weight                        -> blk.33.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.34.attention.wq.weight                    -> blk.34.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.34.attention.wk.weight                    -> blk.34.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.34.attention.wv.weight                    -> blk.34.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.34.attention.wo.weight                    -> blk.34.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.34.feed_forward.w1.weight                 -> blk.34.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.34.feed_forward.w2.weight                 -> blk.34.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.34.feed_forward.w3.weight                 -> blk.34.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.34.attention_norm.weight                  -> blk.34.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.34.ffn_norm.weight                        -> blk.34.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.35.attention.wq.weight                    -> blk.35.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.35.attention.wk.weight                    -> blk.35.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.35.attention.wv.weight                    -> blk.35.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.35.attention.wo.weight                    -> blk.35.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.35.feed_forward.w1.weight                 -> blk.35.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.35.feed_forward.w2.weight                 -> blk.35.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.35.feed_forward.w3.weight                 -> blk.35.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.35.attention_norm.weight                  -> blk.35.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.35.ffn_norm.weight                        -> blk.35.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.36.attention.wq.weight                    -> blk.36.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.36.attention.wk.weight                    -> blk.36.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.36.attention.wv.weight                    -> blk.36.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.36.attention.wo.weight                    -> blk.36.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.36.feed_forward.w1.weight                 -> blk.36.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.36.feed_forward.w2.weight                 -> blk.36.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.36.feed_forward.w3.weight                 -> blk.36.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.36.attention_norm.weight                  -> blk.36.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.36.ffn_norm.weight                        -> blk.36.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.37.attention.wq.weight                    -> blk.37.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.37.attention.wk.weight                    -> blk.37.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.37.attention.wv.weight                    -> blk.37.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.37.attention.wo.weight                    -> blk.37.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.37.feed_forward.w1.weight                 -> blk.37.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.37.feed_forward.w2.weight                 -> blk.37.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.37.feed_forward.w3.weight                 -> blk.37.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.37.attention_norm.weight                  -> blk.37.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.37.ffn_norm.weight                        -> blk.37.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.38.attention.wq.weight                    -> blk.38.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.38.attention.wk.weight                    -> blk.38.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.38.attention.wv.weight                    -> blk.38.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.38.attention.wo.weight                    -> blk.38.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.38.feed_forward.w1.weight                 -> blk.38.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.38.feed_forward.w2.weight                 -> blk.38.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.38.feed_forward.w3.weight                 -> blk.38.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.38.attention_norm.weight                  -> blk.38.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.38.ffn_norm.weight                        -> blk.38.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.39.attention.wq.weight                    -> blk.39.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.39.attention.wk.weight                    -> blk.39.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.39.attention.wv.weight                    -> blk.39.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.39.attention.wo.weight                    -> blk.39.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.39.feed_forward.w1.weight                 -> blk.39.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.39.feed_forward.w2.weight                 -> blk.39.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.39.feed_forward.w3.weight                 -> blk.39.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.39.attention_norm.weight                  -> blk.39.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.39.ffn_norm.weight                        -> blk.39.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.40.attention.wq.weight                    -> blk.40.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.40.attention.wk.weight                    -> blk.40.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.40.attention.wv.weight                    -> blk.40.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.40.attention.wo.weight                    -> blk.40.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.40.feed_forward.w1.weight                 -> blk.40.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.40.feed_forward.w2.weight                 -> blk.40.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.40.feed_forward.w3.weight                 -> blk.40.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.40.attention_norm.weight                  -> blk.40.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.40.ffn_norm.weight                        -> blk.40.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.41.attention.wq.weight                    -> blk.41.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.41.attention.wk.weight                    -> blk.41.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.41.attention.wv.weight                    -> blk.41.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.41.attention.wo.weight                    -> blk.41.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.41.feed_forward.w1.weight                 -> blk.41.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.41.feed_forward.w2.weight                 -> blk.41.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.41.feed_forward.w3.weight                 -> blk.41.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.41.attention_norm.weight                  -> blk.41.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.41.ffn_norm.weight                        -> blk.41.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.42.attention.wq.weight                    -> blk.42.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.42.attention.wk.weight                    -> blk.42.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.42.attention.wv.weight                    -> blk.42.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.42.attention.wo.weight                    -> blk.42.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.42.feed_forward.w1.weight                 -> blk.42.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.42.feed_forward.w2.weight                 -> blk.42.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.42.feed_forward.w3.weight                 -> blk.42.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.42.attention_norm.weight                  -> blk.42.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.42.ffn_norm.weight                        -> blk.42.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.43.attention.wq.weight                    -> blk.43.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.43.attention.wk.weight                    -> blk.43.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.43.attention.wv.weight                    -> blk.43.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.43.attention.wo.weight                    -> blk.43.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.43.feed_forward.w1.weight                 -> blk.43.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.43.feed_forward.w2.weight                 -> blk.43.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.43.feed_forward.w3.weight                 -> blk.43.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.43.attention_norm.weight                  -> blk.43.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.43.ffn_norm.weight                        -> blk.43.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.44.attention.wq.weight                    -> blk.44.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.44.attention.wk.weight                    -> blk.44.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.44.attention.wv.weight                    -> blk.44.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.44.attention.wo.weight                    -> blk.44.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.44.feed_forward.w1.weight                 -> blk.44.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.44.feed_forward.w2.weight                 -> blk.44.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.44.feed_forward.w3.weight                 -> blk.44.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.44.attention_norm.weight                  -> blk.44.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.44.ffn_norm.weight                        -> blk.44.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.45.attention.wq.weight                    -> blk.45.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.45.attention.wk.weight                    -> blk.45.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.45.attention.wv.weight                    -> blk.45.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.45.attention.wo.weight                    -> blk.45.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.45.feed_forward.w1.weight                 -> blk.45.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.45.feed_forward.w2.weight                 -> blk.45.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.45.feed_forward.w3.weight                 -> blk.45.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.45.attention_norm.weight                  -> blk.45.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.45.ffn_norm.weight                        -> blk.45.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.46.attention.wq.weight                    -> blk.46.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.46.attention.wk.weight                    -> blk.46.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.46.attention.wv.weight                    -> blk.46.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.46.attention.wo.weight                    -> blk.46.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.46.feed_forward.w1.weight                 -> blk.46.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.46.feed_forward.w2.weight                 -> blk.46.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.46.feed_forward.w3.weight                 -> blk.46.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.46.attention_norm.weight                  -> blk.46.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.46.ffn_norm.weight                        -> blk.46.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.47.attention.wq.weight                    -> blk.47.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.47.attention.wk.weight                    -> blk.47.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.47.attention.wv.weight                    -> blk.47.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.47.attention.wo.weight                    -> blk.47.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.47.feed_forward.w1.weight                 -> blk.47.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.47.feed_forward.w2.weight                 -> blk.47.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.47.feed_forward.w3.weight                 -> blk.47.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.47.attention_norm.weight                  -> blk.47.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.47.ffn_norm.weight                        -> blk.47.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.48.attention.wq.weight                    -> blk.48.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.48.attention.wk.weight                    -> blk.48.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.48.attention.wv.weight                    -> blk.48.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.48.attention.wo.weight                    -> blk.48.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.48.feed_forward.w1.weight                 -> blk.48.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.48.feed_forward.w2.weight                 -> blk.48.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.48.feed_forward.w3.weight                 -> blk.48.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.48.attention_norm.weight                  -> blk.48.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.48.ffn_norm.weight                        -> blk.48.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.49.attention.wq.weight                    -> blk.49.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.49.attention.wk.weight                    -> blk.49.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.49.attention.wv.weight                    -> blk.49.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.49.attention.wo.weight                    -> blk.49.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.49.feed_forward.w1.weight                 -> blk.49.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.49.feed_forward.w2.weight                 -> blk.49.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.49.feed_forward.w3.weight                 -> blk.49.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.49.attention_norm.weight                  -> blk.49.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.49.ffn_norm.weight                        -> blk.49.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.50.attention.wq.weight                    -> blk.50.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.50.attention.wk.weight                    -> blk.50.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.50.attention.wv.weight                    -> blk.50.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.50.attention.wo.weight                    -> blk.50.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.50.feed_forward.w1.weight                 -> blk.50.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.50.feed_forward.w2.weight                 -> blk.50.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.50.feed_forward.w3.weight                 -> blk.50.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.50.attention_norm.weight                  -> blk.50.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.50.ffn_norm.weight                        -> blk.50.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.51.attention.wq.weight                    -> blk.51.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.51.attention.wk.weight                    -> blk.51.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.51.attention.wv.weight                    -> blk.51.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.51.attention.wo.weight                    -> blk.51.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.51.feed_forward.w1.weight                 -> blk.51.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.51.feed_forward.w2.weight                 -> blk.51.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.51.feed_forward.w3.weight                 -> blk.51.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.51.attention_norm.weight                  -> blk.51.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.51.ffn_norm.weight                        -> blk.51.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.52.attention.wq.weight                    -> blk.52.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.52.attention.wk.weight                    -> blk.52.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.52.attention.wv.weight                    -> blk.52.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.52.attention.wo.weight                    -> blk.52.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.52.feed_forward.w1.weight                 -> blk.52.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.52.feed_forward.w2.weight                 -> blk.52.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.52.feed_forward.w3.weight                 -> blk.52.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.52.attention_norm.weight                  -> blk.52.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.52.ffn_norm.weight                        -> blk.52.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.53.attention.wq.weight                    -> blk.53.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.53.attention.wk.weight                    -> blk.53.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.53.attention.wv.weight                    -> blk.53.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.53.attention.wo.weight                    -> blk.53.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.53.feed_forward.w1.weight                 -> blk.53.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.53.feed_forward.w2.weight                 -> blk.53.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.53.feed_forward.w3.weight                 -> blk.53.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.53.attention_norm.weight                  -> blk.53.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.53.ffn_norm.weight                        -> blk.53.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.54.attention.wq.weight                    -> blk.54.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.54.attention.wk.weight                    -> blk.54.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.54.attention.wv.weight                    -> blk.54.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.54.attention.wo.weight                    -> blk.54.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.54.feed_forward.w1.weight                 -> blk.54.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.54.feed_forward.w2.weight                 -> blk.54.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.54.feed_forward.w3.weight                 -> blk.54.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.54.attention_norm.weight                  -> blk.54.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.54.ffn_norm.weight                        -> blk.54.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.55.attention.wq.weight                    -> blk.55.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.55.attention.wk.weight                    -> blk.55.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.55.attention.wv.weight                    -> blk.55.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.55.attention.wo.weight                    -> blk.55.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.55.feed_forward.w1.weight                 -> blk.55.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.55.feed_forward.w2.weight                 -> blk.55.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.55.feed_forward.w3.weight                 -> blk.55.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.55.attention_norm.weight                  -> blk.55.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.55.ffn_norm.weight                        -> blk.55.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.56.attention.wq.weight                    -> blk.56.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.56.attention.wk.weight                    -> blk.56.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.56.attention.wv.weight                    -> blk.56.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.56.attention.wo.weight                    -> blk.56.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.56.feed_forward.w1.weight                 -> blk.56.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.56.feed_forward.w2.weight                 -> blk.56.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.56.feed_forward.w3.weight                 -> blk.56.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.56.attention_norm.weight                  -> blk.56.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.56.ffn_norm.weight                        -> blk.56.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.57.attention.wq.weight                    -> blk.57.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.57.attention.wk.weight                    -> blk.57.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.57.attention.wv.weight                    -> blk.57.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.57.attention.wo.weight                    -> blk.57.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.57.feed_forward.w1.weight                 -> blk.57.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.57.feed_forward.w2.weight                 -> blk.57.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.57.feed_forward.w3.weight                 -> blk.57.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.57.attention_norm.weight                  -> blk.57.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.57.ffn_norm.weight                        -> blk.57.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.58.attention.wq.weight                    -> blk.58.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.58.attention.wk.weight                    -> blk.58.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.58.attention.wv.weight                    -> blk.58.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.58.attention.wo.weight                    -> blk.58.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.58.feed_forward.w1.weight                 -> blk.58.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.58.feed_forward.w2.weight                 -> blk.58.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.58.feed_forward.w3.weight                 -> blk.58.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.58.attention_norm.weight                  -> blk.58.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.58.ffn_norm.weight                        -> blk.58.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.59.attention.wq.weight                    -> blk.59.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.59.attention.wk.weight                    -> blk.59.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.59.attention.wv.weight                    -> blk.59.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.59.attention.wo.weight                    -> blk.59.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.59.feed_forward.w1.weight                 -> blk.59.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.59.feed_forward.w2.weight                 -> blk.59.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.59.feed_forward.w3.weight                 -> blk.59.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.59.attention_norm.weight                  -> blk.59.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.59.ffn_norm.weight                        -> blk.59.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.60.attention.wq.weight                    -> blk.60.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.60.attention.wk.weight                    -> blk.60.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.60.attention.wv.weight                    -> blk.60.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.60.attention.wo.weight                    -> blk.60.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.60.feed_forward.w1.weight                 -> blk.60.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.60.feed_forward.w2.weight                 -> blk.60.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.60.feed_forward.w3.weight                 -> blk.60.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.60.attention_norm.weight                  -> blk.60.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.60.ffn_norm.weight                        -> blk.60.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.61.attention.wq.weight                    -> blk.61.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.61.attention.wk.weight                    -> blk.61.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.61.attention.wv.weight                    -> blk.61.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.61.attention.wo.weight                    -> blk.61.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.61.feed_forward.w1.weight                 -> blk.61.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.61.feed_forward.w2.weight                 -> blk.61.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.61.feed_forward.w3.weight                 -> blk.61.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.61.attention_norm.weight                  -> blk.61.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.61.ffn_norm.weight                        -> blk.61.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.62.attention.wq.weight                    -> blk.62.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.62.attention.wk.weight                    -> blk.62.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.62.attention.wv.weight                    -> blk.62.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.62.attention.wo.weight                    -> blk.62.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.62.feed_forward.w1.weight                 -> blk.62.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.62.feed_forward.w2.weight                 -> blk.62.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.62.feed_forward.w3.weight                 -> blk.62.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.62.attention_norm.weight                  -> blk.62.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.62.ffn_norm.weight                        -> blk.62.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.63.attention.wq.weight                    -> blk.63.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.63.attention.wk.weight                    -> blk.63.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.63.attention.wv.weight                    -> blk.63.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.63.attention.wo.weight                    -> blk.63.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.63.feed_forward.w1.weight                 -> blk.63.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.63.feed_forward.w2.weight                 -> blk.63.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.63.feed_forward.w3.weight                 -> blk.63.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.63.attention_norm.weight                  -> blk.63.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.63.ffn_norm.weight                        -> blk.63.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.64.attention.wq.weight                    -> blk.64.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.64.attention.wk.weight                    -> blk.64.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.64.attention.wv.weight                    -> blk.64.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.64.attention.wo.weight                    -> blk.64.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.64.feed_forward.w1.weight                 -> blk.64.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.64.feed_forward.w2.weight                 -> blk.64.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.64.feed_forward.w3.weight                 -> blk.64.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.64.attention_norm.weight                  -> blk.64.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.64.ffn_norm.weight                        -> blk.64.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.65.attention.wq.weight                    -> blk.65.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.65.attention.wk.weight                    -> blk.65.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.65.attention.wv.weight                    -> blk.65.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.65.attention.wo.weight                    -> blk.65.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.65.feed_forward.w1.weight                 -> blk.65.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.65.feed_forward.w2.weight                 -> blk.65.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.65.feed_forward.w3.weight                 -> blk.65.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.65.attention_norm.weight                  -> blk.65.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.65.ffn_norm.weight                        -> blk.65.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.66.attention.wq.weight                    -> blk.66.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.66.attention.wk.weight                    -> blk.66.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.66.attention.wv.weight                    -> blk.66.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.66.attention.wo.weight                    -> blk.66.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.66.feed_forward.w1.weight                 -> blk.66.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.66.feed_forward.w2.weight                 -> blk.66.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.66.feed_forward.w3.weight                 -> blk.66.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.66.attention_norm.weight                  -> blk.66.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.66.ffn_norm.weight                        -> blk.66.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.67.attention.wq.weight                    -> blk.67.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.67.attention.wk.weight                    -> blk.67.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.67.attention.wv.weight                    -> blk.67.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.67.attention.wo.weight                    -> blk.67.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.67.feed_forward.w1.weight                 -> blk.67.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.67.feed_forward.w2.weight                 -> blk.67.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.67.feed_forward.w3.weight                 -> blk.67.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.67.attention_norm.weight                  -> blk.67.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.67.ffn_norm.weight                        -> blk.67.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.68.attention.wq.weight                    -> blk.68.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.68.attention.wk.weight                    -> blk.68.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.68.attention.wv.weight                    -> blk.68.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.68.attention.wo.weight                    -> blk.68.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.68.feed_forward.w1.weight                 -> blk.68.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.68.feed_forward.w2.weight                 -> blk.68.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.68.feed_forward.w3.weight                 -> blk.68.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.68.attention_norm.weight                  -> blk.68.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.68.ffn_norm.weight                        -> blk.68.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.69.attention.wq.weight                    -> blk.69.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.69.attention.wk.weight                    -> blk.69.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.69.attention.wv.weight                    -> blk.69.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.69.attention.wo.weight                    -> blk.69.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.69.feed_forward.w1.weight                 -> blk.69.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.69.feed_forward.w2.weight                 -> blk.69.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.69.feed_forward.w3.weight                 -> blk.69.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.69.attention_norm.weight                  -> blk.69.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.69.ffn_norm.weight                        -> blk.69.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.70.attention.wq.weight                    -> blk.70.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.70.attention.wk.weight                    -> blk.70.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.70.attention.wv.weight                    -> blk.70.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.70.attention.wo.weight                    -> blk.70.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.70.feed_forward.w1.weight                 -> blk.70.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.70.feed_forward.w2.weight                 -> blk.70.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.70.feed_forward.w3.weight                 -> blk.70.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.70.attention_norm.weight                  -> blk.70.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.70.ffn_norm.weight                        -> blk.70.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.71.attention.wq.weight                    -> blk.71.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.71.attention.wk.weight                    -> blk.71.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.71.attention.wv.weight                    -> blk.71.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.71.attention.wo.weight                    -> blk.71.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.71.feed_forward.w1.weight                 -> blk.71.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.71.feed_forward.w2.weight                 -> blk.71.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.71.feed_forward.w3.weight                 -> blk.71.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.71.attention_norm.weight                  -> blk.71.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.71.ffn_norm.weight                        -> blk.71.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.72.attention.wq.weight                    -> blk.72.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.72.attention.wk.weight                    -> blk.72.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.72.attention.wv.weight                    -> blk.72.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.72.attention.wo.weight                    -> blk.72.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.72.feed_forward.w1.weight                 -> blk.72.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.72.feed_forward.w2.weight                 -> blk.72.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.72.feed_forward.w3.weight                 -> blk.72.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.72.attention_norm.weight                  -> blk.72.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.72.ffn_norm.weight                        -> blk.72.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.73.attention.wq.weight                    -> blk.73.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.73.attention.wk.weight                    -> blk.73.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.73.attention.wv.weight                    -> blk.73.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.73.attention.wo.weight                    -> blk.73.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.73.feed_forward.w1.weight                 -> blk.73.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.73.feed_forward.w2.weight                 -> blk.73.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.73.feed_forward.w3.weight                 -> blk.73.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.73.attention_norm.weight                  -> blk.73.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.73.ffn_norm.weight                        -> blk.73.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.74.attention.wq.weight                    -> blk.74.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.74.attention.wk.weight                    -> blk.74.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.74.attention.wv.weight                    -> blk.74.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.74.attention.wo.weight                    -> blk.74.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.74.feed_forward.w1.weight                 -> blk.74.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.74.feed_forward.w2.weight                 -> blk.74.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.74.feed_forward.w3.weight                 -> blk.74.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.74.attention_norm.weight                  -> blk.74.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.74.ffn_norm.weight                        -> blk.74.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.75.attention.wq.weight                    -> blk.75.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.75.attention.wk.weight                    -> blk.75.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.75.attention.wv.weight                    -> blk.75.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.75.attention.wo.weight                    -> blk.75.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.75.feed_forward.w1.weight                 -> blk.75.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.75.feed_forward.w2.weight                 -> blk.75.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.75.feed_forward.w3.weight                 -> blk.75.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.75.attention_norm.weight                  -> blk.75.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.75.ffn_norm.weight                        -> blk.75.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.76.attention.wq.weight                    -> blk.76.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.76.attention.wk.weight                    -> blk.76.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.76.attention.wv.weight                    -> blk.76.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.76.attention.wo.weight                    -> blk.76.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.76.feed_forward.w1.weight                 -> blk.76.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.76.feed_forward.w2.weight                 -> blk.76.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.76.feed_forward.w3.weight                 -> blk.76.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.76.attention_norm.weight                  -> blk.76.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.76.ffn_norm.weight                        -> blk.76.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.77.attention.wq.weight                    -> blk.77.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.77.attention.wk.weight                    -> blk.77.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.77.attention.wv.weight                    -> blk.77.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.77.attention.wo.weight                    -> blk.77.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.77.feed_forward.w1.weight                 -> blk.77.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.77.feed_forward.w2.weight                 -> blk.77.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.77.feed_forward.w3.weight                 -> blk.77.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.77.attention_norm.weight                  -> blk.77.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.77.ffn_norm.weight                        -> blk.77.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.78.attention.wq.weight                    -> blk.78.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.78.attention.wk.weight                    -> blk.78.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.78.attention.wv.weight                    -> blk.78.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.78.attention.wo.weight                    -> blk.78.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.78.feed_forward.w1.weight                 -> blk.78.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.78.feed_forward.w2.weight                 -> blk.78.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.78.feed_forward.w3.weight                 -> blk.78.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.78.attention_norm.weight                  -> blk.78.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.78.ffn_norm.weight                        -> blk.78.ffn_norm.weight                   | BF16   | [8192]\n",
      "layers.79.attention.wq.weight                    -> blk.79.attn_q.weight                     | BF16   | [8192, 8192]\n",
      "layers.79.attention.wk.weight                    -> blk.79.attn_k.weight                     | BF16   | [1024, 8192]\n",
      "layers.79.attention.wv.weight                    -> blk.79.attn_v.weight                     | BF16   | [1024, 8192]\n",
      "layers.79.attention.wo.weight                    -> blk.79.attn_output.weight                | BF16   | [8192, 8192]\n",
      "layers.79.feed_forward.w1.weight                 -> blk.79.ffn_gate.weight                   | BF16   | [28672, 8192]\n",
      "layers.79.feed_forward.w2.weight                 -> blk.79.ffn_down.weight                   | BF16   | [8192, 28672]\n",
      "layers.79.feed_forward.w3.weight                 -> blk.79.ffn_up.weight                     | BF16   | [28672, 8192]\n",
      "layers.79.attention_norm.weight                  -> blk.79.attn_norm.weight                  | BF16   | [8192]\n",
      "layers.79.ffn_norm.weight                        -> blk.79.ffn_norm.weight                   | BF16   | [8192]\n",
      "skipping tensor rope_freqs\n",
      "Writing models/70B-v2/ggml-model-f16.gguf, format 1\n",
      "gguf: This GGUF file is for Little Endian only\n",
      "gguf: Adding 61249 merge(s).\n",
      "gguf: Setting special token type bos to 1\n",
      "gguf: Setting special token type eos to 2\n",
      "gguf: Setting special token type unk to 0\n",
      "gguf: Setting add_bos_token to True\n",
      "gguf: Setting add_eos_token to False\n",
      "[  1/723] Writing tensor token_embd.weight                      | size  32000 x   8192  | type F16  | T+   2\n",
      "[  2/723] Writing tensor output_norm.weight                     | size   8192           | type F32  | T+   2\n",
      "[  3/723] Writing tensor output.weight                          | size  32000 x   8192  | type F16  | T+   2\n",
      "[  4/723] Writing tensor blk.0.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   3\n",
      "[  5/723] Writing tensor blk.0.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   3\n",
      "[  6/723] Writing tensor blk.0.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   3\n",
      "[  7/723] Writing tensor blk.0.attn_output.weight               | size   8192 x   8192  | type F16  | T+   3\n",
      "[  8/723] Writing tensor blk.0.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+   3\n",
      "[  9/723] Writing tensor blk.0.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+   4\n",
      "[ 10/723] Writing tensor blk.0.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+   5\n",
      "[ 11/723] Writing tensor blk.0.attn_norm.weight                 | size   8192           | type F32  | T+   5\n",
      "[ 12/723] Writing tensor blk.0.ffn_norm.weight                  | size   8192           | type F32  | T+   5\n",
      "[ 13/723] Writing tensor blk.1.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   5\n",
      "[ 14/723] Writing tensor blk.1.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   5\n",
      "[ 15/723] Writing tensor blk.1.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   5\n",
      "[ 16/723] Writing tensor blk.1.attn_output.weight               | size   8192 x   8192  | type F16  | T+   5\n",
      "[ 17/723] Writing tensor blk.1.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+   9\n",
      "[ 18/723] Writing tensor blk.1.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+   9\n",
      "[ 19/723] Writing tensor blk.1.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+   9\n",
      "[ 20/723] Writing tensor blk.1.attn_norm.weight                 | size   8192           | type F32  | T+   9\n",
      "[ 21/723] Writing tensor blk.1.ffn_norm.weight                  | size   8192           | type F32  | T+   9\n",
      "[ 22/723] Writing tensor blk.2.attn_q.weight                    | size   8192 x   8192  | type F16  | T+   9\n",
      "[ 23/723] Writing tensor blk.2.attn_k.weight                    | size   1024 x   8192  | type F16  | T+   9\n",
      "[ 24/723] Writing tensor blk.2.attn_v.weight                    | size   1024 x   8192  | type F16  | T+   9\n",
      "[ 25/723] Writing tensor blk.2.attn_output.weight               | size   8192 x   8192  | type F16  | T+   9\n",
      "[ 26/723] Writing tensor blk.2.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  11\n",
      "[ 27/723] Writing tensor blk.2.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  11\n",
      "[ 28/723] Writing tensor blk.2.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  11\n",
      "[ 29/723] Writing tensor blk.2.attn_norm.weight                 | size   8192           | type F32  | T+  12\n",
      "[ 30/723] Writing tensor blk.2.ffn_norm.weight                  | size   8192           | type F32  | T+  12\n",
      "[ 31/723] Writing tensor blk.3.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  12\n",
      "[ 32/723] Writing tensor blk.3.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  12\n",
      "[ 33/723] Writing tensor blk.3.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  12\n",
      "[ 34/723] Writing tensor blk.3.attn_output.weight               | size   8192 x   8192  | type F16  | T+  12\n",
      "[ 35/723] Writing tensor blk.3.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  13\n",
      "[ 36/723] Writing tensor blk.3.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  14\n",
      "[ 37/723] Writing tensor blk.3.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  14\n",
      "[ 38/723] Writing tensor blk.3.attn_norm.weight                 | size   8192           | type F32  | T+  14\n",
      "[ 39/723] Writing tensor blk.3.ffn_norm.weight                  | size   8192           | type F32  | T+  14\n",
      "[ 40/723] Writing tensor blk.4.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  14\n",
      "[ 41/723] Writing tensor blk.4.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  14\n",
      "[ 42/723] Writing tensor blk.4.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  14\n",
      "[ 43/723] Writing tensor blk.4.attn_output.weight               | size   8192 x   8192  | type F16  | T+  14\n",
      "[ 44/723] Writing tensor blk.4.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  16\n",
      "[ 45/723] Writing tensor blk.4.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  16\n",
      "[ 46/723] Writing tensor blk.4.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  16\n",
      "[ 47/723] Writing tensor blk.4.attn_norm.weight                 | size   8192           | type F32  | T+  16\n",
      "[ 48/723] Writing tensor blk.4.ffn_norm.weight                  | size   8192           | type F32  | T+  16\n",
      "[ 49/723] Writing tensor blk.5.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  16\n",
      "[ 50/723] Writing tensor blk.5.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  16\n",
      "[ 51/723] Writing tensor blk.5.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  16\n",
      "[ 52/723] Writing tensor blk.5.attn_output.weight               | size   8192 x   8192  | type F16  | T+  16\n",
      "[ 53/723] Writing tensor blk.5.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  18\n",
      "[ 54/723] Writing tensor blk.5.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  18\n",
      "[ 55/723] Writing tensor blk.5.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  19\n",
      "[ 56/723] Writing tensor blk.5.attn_norm.weight                 | size   8192           | type F32  | T+  19\n",
      "[ 57/723] Writing tensor blk.5.ffn_norm.weight                  | size   8192           | type F32  | T+  19\n",
      "[ 58/723] Writing tensor blk.6.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  19\n",
      "[ 59/723] Writing tensor blk.6.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  19\n",
      "[ 60/723] Writing tensor blk.6.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  19\n",
      "[ 61/723] Writing tensor blk.6.attn_output.weight               | size   8192 x   8192  | type F16  | T+  19\n",
      "[ 62/723] Writing tensor blk.6.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  21\n",
      "[ 63/723] Writing tensor blk.6.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  21\n",
      "[ 64/723] Writing tensor blk.6.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  21\n",
      "[ 65/723] Writing tensor blk.6.attn_norm.weight                 | size   8192           | type F32  | T+  21\n",
      "[ 66/723] Writing tensor blk.6.ffn_norm.weight                  | size   8192           | type F32  | T+  21\n",
      "[ 67/723] Writing tensor blk.7.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  21\n",
      "[ 68/723] Writing tensor blk.7.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  21\n",
      "[ 69/723] Writing tensor blk.7.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  21\n",
      "[ 70/723] Writing tensor blk.7.attn_output.weight               | size   8192 x   8192  | type F16  | T+  21\n",
      "[ 71/723] Writing tensor blk.7.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  23\n",
      "[ 72/723] Writing tensor blk.7.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  23\n",
      "[ 73/723] Writing tensor blk.7.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  24\n",
      "[ 74/723] Writing tensor blk.7.attn_norm.weight                 | size   8192           | type F32  | T+  24\n",
      "[ 75/723] Writing tensor blk.7.ffn_norm.weight                  | size   8192           | type F32  | T+  24\n",
      "[ 76/723] Writing tensor blk.8.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  24\n",
      "[ 77/723] Writing tensor blk.8.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  24\n",
      "[ 78/723] Writing tensor blk.8.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  24\n",
      "[ 79/723] Writing tensor blk.8.attn_output.weight               | size   8192 x   8192  | type F16  | T+  24\n",
      "[ 80/723] Writing tensor blk.8.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  26\n",
      "[ 81/723] Writing tensor blk.8.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  26\n",
      "[ 82/723] Writing tensor blk.8.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  26\n",
      "[ 83/723] Writing tensor blk.8.attn_norm.weight                 | size   8192           | type F32  | T+  26\n",
      "[ 84/723] Writing tensor blk.8.ffn_norm.weight                  | size   8192           | type F32  | T+  26\n",
      "[ 85/723] Writing tensor blk.9.attn_q.weight                    | size   8192 x   8192  | type F16  | T+  26\n",
      "[ 86/723] Writing tensor blk.9.attn_k.weight                    | size   1024 x   8192  | type F16  | T+  26\n",
      "[ 87/723] Writing tensor blk.9.attn_v.weight                    | size   1024 x   8192  | type F16  | T+  26\n",
      "[ 88/723] Writing tensor blk.9.attn_output.weight               | size   8192 x   8192  | type F16  | T+  26\n",
      "[ 89/723] Writing tensor blk.9.ffn_gate.weight                  | size  28672 x   8192  | type F16  | T+  28\n",
      "[ 90/723] Writing tensor blk.9.ffn_down.weight                  | size   8192 x  28672  | type F16  | T+  28\n",
      "[ 91/723] Writing tensor blk.9.ffn_up.weight                    | size  28672 x   8192  | type F16  | T+  29\n",
      "[ 92/723] Writing tensor blk.9.attn_norm.weight                 | size   8192           | type F32  | T+  29\n",
      "[ 93/723] Writing tensor blk.9.ffn_norm.weight                  | size   8192           | type F32  | T+  29\n",
      "[ 94/723] Writing tensor blk.10.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  29\n",
      "[ 95/723] Writing tensor blk.10.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  29\n",
      "[ 96/723] Writing tensor blk.10.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  29\n",
      "[ 97/723] Writing tensor blk.10.attn_output.weight              | size   8192 x   8192  | type F16  | T+  29\n",
      "[ 98/723] Writing tensor blk.10.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  31\n",
      "[ 99/723] Writing tensor blk.10.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  31\n",
      "[100/723] Writing tensor blk.10.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  31\n",
      "[101/723] Writing tensor blk.10.attn_norm.weight                | size   8192           | type F32  | T+  31\n",
      "[102/723] Writing tensor blk.10.ffn_norm.weight                 | size   8192           | type F32  | T+  31\n",
      "[103/723] Writing tensor blk.11.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  31\n",
      "[104/723] Writing tensor blk.11.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  32\n",
      "[105/723] Writing tensor blk.11.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  32\n",
      "[106/723] Writing tensor blk.11.attn_output.weight              | size   8192 x   8192  | type F16  | T+  32\n",
      "[107/723] Writing tensor blk.11.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  36\n",
      "[108/723] Writing tensor blk.11.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  36\n",
      "[109/723] Writing tensor blk.11.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  37\n",
      "[110/723] Writing tensor blk.11.attn_norm.weight                | size   8192           | type F32  | T+  37\n",
      "[111/723] Writing tensor blk.11.ffn_norm.weight                 | size   8192           | type F32  | T+  37\n",
      "[112/723] Writing tensor blk.12.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  37\n",
      "[113/723] Writing tensor blk.12.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  37\n",
      "[114/723] Writing tensor blk.12.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  37\n",
      "[115/723] Writing tensor blk.12.attn_output.weight              | size   8192 x   8192  | type F16  | T+  37\n",
      "[116/723] Writing tensor blk.12.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  39\n",
      "[117/723] Writing tensor blk.12.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  39\n",
      "[118/723] Writing tensor blk.12.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  39\n",
      "[119/723] Writing tensor blk.12.attn_norm.weight                | size   8192           | type F32  | T+  39\n",
      "[120/723] Writing tensor blk.12.ffn_norm.weight                 | size   8192           | type F32  | T+  39\n",
      "[121/723] Writing tensor blk.13.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  39\n",
      "[122/723] Writing tensor blk.13.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  39\n",
      "[123/723] Writing tensor blk.13.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  39\n",
      "[124/723] Writing tensor blk.13.attn_output.weight              | size   8192 x   8192  | type F16  | T+  39\n",
      "[125/723] Writing tensor blk.13.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  41\n",
      "[126/723] Writing tensor blk.13.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  41\n",
      "[127/723] Writing tensor blk.13.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  42\n",
      "[128/723] Writing tensor blk.13.attn_norm.weight                | size   8192           | type F32  | T+  42\n",
      "[129/723] Writing tensor blk.13.ffn_norm.weight                 | size   8192           | type F32  | T+  42\n",
      "[130/723] Writing tensor blk.14.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  42\n",
      "[131/723] Writing tensor blk.14.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  42\n",
      "[132/723] Writing tensor blk.14.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  42\n",
      "[133/723] Writing tensor blk.14.attn_output.weight              | size   8192 x   8192  | type F16  | T+  42\n",
      "[134/723] Writing tensor blk.14.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  44\n",
      "[135/723] Writing tensor blk.14.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  44\n",
      "[136/723] Writing tensor blk.14.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  44\n",
      "[137/723] Writing tensor blk.14.attn_norm.weight                | size   8192           | type F32  | T+  44\n",
      "[138/723] Writing tensor blk.14.ffn_norm.weight                 | size   8192           | type F32  | T+  44\n",
      "[139/723] Writing tensor blk.15.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  44\n",
      "[140/723] Writing tensor blk.15.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  44\n",
      "[141/723] Writing tensor blk.15.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  44\n",
      "[142/723] Writing tensor blk.15.attn_output.weight              | size   8192 x   8192  | type F16  | T+  44\n",
      "[143/723] Writing tensor blk.15.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  46\n",
      "[144/723] Writing tensor blk.15.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  46\n",
      "[145/723] Writing tensor blk.15.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  47\n",
      "[146/723] Writing tensor blk.15.attn_norm.weight                | size   8192           | type F32  | T+  47\n",
      "[147/723] Writing tensor blk.15.ffn_norm.weight                 | size   8192           | type F32  | T+  47\n",
      "[148/723] Writing tensor blk.16.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  47\n",
      "[149/723] Writing tensor blk.16.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  47\n",
      "[150/723] Writing tensor blk.16.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  47\n",
      "[151/723] Writing tensor blk.16.attn_output.weight              | size   8192 x   8192  | type F16  | T+  47\n",
      "[152/723] Writing tensor blk.16.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  49\n",
      "[153/723] Writing tensor blk.16.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  49\n",
      "[154/723] Writing tensor blk.16.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  49\n",
      "[155/723] Writing tensor blk.16.attn_norm.weight                | size   8192           | type F32  | T+  49\n",
      "[156/723] Writing tensor blk.16.ffn_norm.weight                 | size   8192           | type F32  | T+  49\n",
      "[157/723] Writing tensor blk.17.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  49\n",
      "[158/723] Writing tensor blk.17.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  49\n",
      "[159/723] Writing tensor blk.17.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  49\n",
      "[160/723] Writing tensor blk.17.attn_output.weight              | size   8192 x   8192  | type F16  | T+  49\n",
      "[161/723] Writing tensor blk.17.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  51\n",
      "[162/723] Writing tensor blk.17.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  51\n",
      "[163/723] Writing tensor blk.17.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  51\n",
      "[164/723] Writing tensor blk.17.attn_norm.weight                | size   8192           | type F32  | T+  52\n",
      "[165/723] Writing tensor blk.17.ffn_norm.weight                 | size   8192           | type F32  | T+  52\n",
      "[166/723] Writing tensor blk.18.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  52\n",
      "[167/723] Writing tensor blk.18.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  52\n",
      "[168/723] Writing tensor blk.18.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  52\n",
      "[169/723] Writing tensor blk.18.attn_output.weight              | size   8192 x   8192  | type F16  | T+  52\n",
      "[170/723] Writing tensor blk.18.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  53\n",
      "[171/723] Writing tensor blk.18.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  54\n",
      "[172/723] Writing tensor blk.18.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  54\n",
      "[173/723] Writing tensor blk.18.attn_norm.weight                | size   8192           | type F32  | T+  54\n",
      "[174/723] Writing tensor blk.18.ffn_norm.weight                 | size   8192           | type F32  | T+  54\n",
      "[175/723] Writing tensor blk.19.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  54\n",
      "[176/723] Writing tensor blk.19.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  54\n",
      "[177/723] Writing tensor blk.19.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  54\n",
      "[178/723] Writing tensor blk.19.attn_output.weight              | size   8192 x   8192  | type F16  | T+  54\n",
      "[179/723] Writing tensor blk.19.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  56\n",
      "[180/723] Writing tensor blk.19.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  56\n",
      "[181/723] Writing tensor blk.19.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  56\n",
      "[182/723] Writing tensor blk.19.attn_norm.weight                | size   8192           | type F32  | T+  57\n",
      "[183/723] Writing tensor blk.19.ffn_norm.weight                 | size   8192           | type F32  | T+  57\n",
      "[184/723] Writing tensor blk.20.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  57\n",
      "[185/723] Writing tensor blk.20.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  57\n",
      "[186/723] Writing tensor blk.20.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  57\n",
      "[187/723] Writing tensor blk.20.attn_output.weight              | size   8192 x   8192  | type F16  | T+  57\n",
      "[188/723] Writing tensor blk.20.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  58\n",
      "[189/723] Writing tensor blk.20.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  59\n",
      "[190/723] Writing tensor blk.20.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  59\n",
      "[191/723] Writing tensor blk.20.attn_norm.weight                | size   8192           | type F32  | T+  59\n",
      "[192/723] Writing tensor blk.20.ffn_norm.weight                 | size   8192           | type F32  | T+  59\n",
      "[193/723] Writing tensor blk.21.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  59\n",
      "[194/723] Writing tensor blk.21.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  59\n",
      "[195/723] Writing tensor blk.21.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  59\n",
      "[196/723] Writing tensor blk.21.attn_output.weight              | size   8192 x   8192  | type F16  | T+  59\n",
      "[197/723] Writing tensor blk.21.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  61\n",
      "[198/723] Writing tensor blk.21.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  61\n",
      "[199/723] Writing tensor blk.21.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  61\n",
      "[200/723] Writing tensor blk.21.attn_norm.weight                | size   8192           | type F32  | T+  61\n",
      "[201/723] Writing tensor blk.21.ffn_norm.weight                 | size   8192           | type F32  | T+  61\n",
      "[202/723] Writing tensor blk.22.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  61\n",
      "[203/723] Writing tensor blk.22.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  62\n",
      "[204/723] Writing tensor blk.22.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  62\n",
      "[205/723] Writing tensor blk.22.attn_output.weight              | size   8192 x   8192  | type F16  | T+  62\n",
      "[206/723] Writing tensor blk.22.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  66\n",
      "[207/723] Writing tensor blk.22.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  66\n",
      "[208/723] Writing tensor blk.22.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  67\n",
      "[209/723] Writing tensor blk.22.attn_norm.weight                | size   8192           | type F32  | T+  67\n",
      "[210/723] Writing tensor blk.22.ffn_norm.weight                 | size   8192           | type F32  | T+  67\n",
      "[211/723] Writing tensor blk.23.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  67\n",
      "[212/723] Writing tensor blk.23.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  67\n",
      "[213/723] Writing tensor blk.23.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  67\n",
      "[214/723] Writing tensor blk.23.attn_output.weight              | size   8192 x   8192  | type F16  | T+  67\n",
      "[215/723] Writing tensor blk.23.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  69\n",
      "[216/723] Writing tensor blk.23.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  69\n",
      "[217/723] Writing tensor blk.23.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  70\n",
      "[218/723] Writing tensor blk.23.attn_norm.weight                | size   8192           | type F32  | T+  70\n",
      "[219/723] Writing tensor blk.23.ffn_norm.weight                 | size   8192           | type F32  | T+  70\n",
      "[220/723] Writing tensor blk.24.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  70\n",
      "[221/723] Writing tensor blk.24.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  70\n",
      "[222/723] Writing tensor blk.24.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  70\n",
      "[223/723] Writing tensor blk.24.attn_output.weight              | size   8192 x   8192  | type F16  | T+  70\n",
      "[224/723] Writing tensor blk.24.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  72\n",
      "[225/723] Writing tensor blk.24.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  72\n",
      "[226/723] Writing tensor blk.24.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  72\n",
      "[227/723] Writing tensor blk.24.attn_norm.weight                | size   8192           | type F32  | T+  72\n",
      "[228/723] Writing tensor blk.24.ffn_norm.weight                 | size   8192           | type F32  | T+  72\n",
      "[229/723] Writing tensor blk.25.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  72\n",
      "[230/723] Writing tensor blk.25.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  72\n",
      "[231/723] Writing tensor blk.25.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  72\n",
      "[232/723] Writing tensor blk.25.attn_output.weight              | size   8192 x   8192  | type F16  | T+  72\n",
      "[233/723] Writing tensor blk.25.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  74\n",
      "[234/723] Writing tensor blk.25.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  75\n",
      "[235/723] Writing tensor blk.25.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  75\n",
      "[236/723] Writing tensor blk.25.attn_norm.weight                | size   8192           | type F32  | T+  75\n",
      "[237/723] Writing tensor blk.25.ffn_norm.weight                 | size   8192           | type F32  | T+  75\n",
      "[238/723] Writing tensor blk.26.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  75\n",
      "[239/723] Writing tensor blk.26.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  75\n",
      "[240/723] Writing tensor blk.26.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  75\n",
      "[241/723] Writing tensor blk.26.attn_output.weight              | size   8192 x   8192  | type F16  | T+  75\n",
      "[242/723] Writing tensor blk.26.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  77\n",
      "[243/723] Writing tensor blk.26.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  77\n",
      "[244/723] Writing tensor blk.26.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  77\n",
      "[245/723] Writing tensor blk.26.attn_norm.weight                | size   8192           | type F32  | T+  78\n",
      "[246/723] Writing tensor blk.26.ffn_norm.weight                 | size   8192           | type F32  | T+  78\n",
      "[247/723] Writing tensor blk.27.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  78\n",
      "[248/723] Writing tensor blk.27.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  78\n",
      "[249/723] Writing tensor blk.27.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  78\n",
      "[250/723] Writing tensor blk.27.attn_output.weight              | size   8192 x   8192  | type F16  | T+  78\n",
      "[251/723] Writing tensor blk.27.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  80\n",
      "[252/723] Writing tensor blk.27.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  80\n",
      "[253/723] Writing tensor blk.27.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  80\n",
      "[254/723] Writing tensor blk.27.attn_norm.weight                | size   8192           | type F32  | T+  80\n",
      "[255/723] Writing tensor blk.27.ffn_norm.weight                 | size   8192           | type F32  | T+  80\n",
      "[256/723] Writing tensor blk.28.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  80\n",
      "[257/723] Writing tensor blk.28.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  80\n",
      "[258/723] Writing tensor blk.28.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  80\n",
      "[259/723] Writing tensor blk.28.attn_output.weight              | size   8192 x   8192  | type F16  | T+  80\n",
      "[260/723] Writing tensor blk.28.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  82\n",
      "[261/723] Writing tensor blk.28.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  82\n",
      "[262/723] Writing tensor blk.28.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  82\n",
      "[263/723] Writing tensor blk.28.attn_norm.weight                | size   8192           | type F32  | T+  83\n",
      "[264/723] Writing tensor blk.28.ffn_norm.weight                 | size   8192           | type F32  | T+  83\n",
      "[265/723] Writing tensor blk.29.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  83\n",
      "[266/723] Writing tensor blk.29.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  83\n",
      "[267/723] Writing tensor blk.29.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  83\n",
      "[268/723] Writing tensor blk.29.attn_output.weight              | size   8192 x   8192  | type F16  | T+  83\n",
      "[269/723] Writing tensor blk.29.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  85\n",
      "[270/723] Writing tensor blk.29.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  85\n",
      "[271/723] Writing tensor blk.29.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  85\n",
      "[272/723] Writing tensor blk.29.attn_norm.weight                | size   8192           | type F32  | T+  85\n",
      "[273/723] Writing tensor blk.29.ffn_norm.weight                 | size   8192           | type F32  | T+  85\n",
      "[274/723] Writing tensor blk.30.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  85\n",
      "[275/723] Writing tensor blk.30.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  85\n",
      "[276/723] Writing tensor blk.30.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  85\n",
      "[277/723] Writing tensor blk.30.attn_output.weight              | size   8192 x   8192  | type F16  | T+  85\n",
      "[278/723] Writing tensor blk.30.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  87\n",
      "[279/723] Writing tensor blk.30.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  87\n",
      "[280/723] Writing tensor blk.30.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  88\n",
      "[281/723] Writing tensor blk.30.attn_norm.weight                | size   8192           | type F32  | T+  88\n",
      "[282/723] Writing tensor blk.30.ffn_norm.weight                 | size   8192           | type F32  | T+  88\n",
      "[283/723] Writing tensor blk.31.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  88\n",
      "[284/723] Writing tensor blk.31.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  88\n",
      "[285/723] Writing tensor blk.31.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  88\n",
      "[286/723] Writing tensor blk.31.attn_output.weight              | size   8192 x   8192  | type F16  | T+  88\n",
      "[287/723] Writing tensor blk.31.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  90\n",
      "[288/723] Writing tensor blk.31.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  90\n",
      "[289/723] Writing tensor blk.31.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  90\n",
      "[290/723] Writing tensor blk.31.attn_norm.weight                | size   8192           | type F32  | T+  90\n",
      "[291/723] Writing tensor blk.31.ffn_norm.weight                 | size   8192           | type F32  | T+  90\n",
      "[292/723] Writing tensor blk.32.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  90\n",
      "[293/723] Writing tensor blk.32.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  90\n",
      "[294/723] Writing tensor blk.32.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  90\n",
      "[295/723] Writing tensor blk.32.attn_output.weight              | size   8192 x   8192  | type F16  | T+  91\n",
      "[296/723] Writing tensor blk.32.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  92\n",
      "[297/723] Writing tensor blk.32.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  93\n",
      "[298/723] Writing tensor blk.32.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  93\n",
      "[299/723] Writing tensor blk.32.attn_norm.weight                | size   8192           | type F32  | T+  93\n",
      "[300/723] Writing tensor blk.32.ffn_norm.weight                 | size   8192           | type F32  | T+  93\n",
      "[301/723] Writing tensor blk.33.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  93\n",
      "[302/723] Writing tensor blk.33.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  93\n",
      "[303/723] Writing tensor blk.33.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  93\n",
      "[304/723] Writing tensor blk.33.attn_output.weight              | size   8192 x   8192  | type F16  | T+  93\n",
      "[305/723] Writing tensor blk.33.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+  95\n",
      "[306/723] Writing tensor blk.33.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+  95\n",
      "[307/723] Writing tensor blk.33.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+  95\n",
      "[308/723] Writing tensor blk.33.attn_norm.weight                | size   8192           | type F32  | T+  96\n",
      "[309/723] Writing tensor blk.33.ffn_norm.weight                 | size   8192           | type F32  | T+  96\n",
      "[310/723] Writing tensor blk.34.attn_q.weight                   | size   8192 x   8192  | type F16  | T+  96\n",
      "[311/723] Writing tensor blk.34.attn_k.weight                   | size   1024 x   8192  | type F16  | T+  96\n",
      "[312/723] Writing tensor blk.34.attn_v.weight                   | size   1024 x   8192  | type F16  | T+  96\n",
      "[313/723] Writing tensor blk.34.attn_output.weight              | size   8192 x   8192  | type F16  | T+  96\n",
      "[314/723] Writing tensor blk.34.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 100\n",
      "[315/723] Writing tensor blk.34.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 100\n",
      "[316/723] Writing tensor blk.34.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 100\n",
      "[317/723] Writing tensor blk.34.attn_norm.weight                | size   8192           | type F32  | T+ 101\n",
      "[318/723] Writing tensor blk.34.ffn_norm.weight                 | size   8192           | type F32  | T+ 101\n",
      "[319/723] Writing tensor blk.35.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 101\n",
      "[320/723] Writing tensor blk.35.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 101\n",
      "[321/723] Writing tensor blk.35.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 101\n",
      "[322/723] Writing tensor blk.35.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 101\n",
      "[323/723] Writing tensor blk.35.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 103\n",
      "[324/723] Writing tensor blk.35.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 103\n",
      "[325/723] Writing tensor blk.35.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 103\n",
      "[326/723] Writing tensor blk.35.attn_norm.weight                | size   8192           | type F32  | T+ 104\n",
      "[327/723] Writing tensor blk.35.ffn_norm.weight                 | size   8192           | type F32  | T+ 104\n",
      "[328/723] Writing tensor blk.36.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 104\n",
      "[329/723] Writing tensor blk.36.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 104\n",
      "[330/723] Writing tensor blk.36.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 104\n",
      "[331/723] Writing tensor blk.36.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 104\n",
      "[332/723] Writing tensor blk.36.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 106\n",
      "[333/723] Writing tensor blk.36.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 106\n",
      "[334/723] Writing tensor blk.36.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 106\n",
      "[335/723] Writing tensor blk.36.attn_norm.weight                | size   8192           | type F32  | T+ 106\n",
      "[336/723] Writing tensor blk.36.ffn_norm.weight                 | size   8192           | type F32  | T+ 106\n",
      "[337/723] Writing tensor blk.37.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 106\n",
      "[338/723] Writing tensor blk.37.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 106\n",
      "[339/723] Writing tensor blk.37.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 106\n",
      "[340/723] Writing tensor blk.37.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 106\n",
      "[341/723] Writing tensor blk.37.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 108\n",
      "[342/723] Writing tensor blk.37.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 108\n",
      "[343/723] Writing tensor blk.37.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 108\n",
      "[344/723] Writing tensor blk.37.attn_norm.weight                | size   8192           | type F32  | T+ 109\n",
      "[345/723] Writing tensor blk.37.ffn_norm.weight                 | size   8192           | type F32  | T+ 109\n",
      "[346/723] Writing tensor blk.38.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 109\n",
      "[347/723] Writing tensor blk.38.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 109\n",
      "[348/723] Writing tensor blk.38.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 109\n",
      "[349/723] Writing tensor blk.38.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 109\n",
      "[350/723] Writing tensor blk.38.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 111\n",
      "[351/723] Writing tensor blk.38.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 111\n",
      "[352/723] Writing tensor blk.38.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 111\n",
      "[353/723] Writing tensor blk.38.attn_norm.weight                | size   8192           | type F32  | T+ 111\n",
      "[354/723] Writing tensor blk.38.ffn_norm.weight                 | size   8192           | type F32  | T+ 111\n",
      "[355/723] Writing tensor blk.39.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 111\n",
      "[356/723] Writing tensor blk.39.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 111\n",
      "[357/723] Writing tensor blk.39.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 111\n",
      "[358/723] Writing tensor blk.39.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 111\n",
      "[359/723] Writing tensor blk.39.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 113\n",
      "[360/723] Writing tensor blk.39.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 113\n",
      "[361/723] Writing tensor blk.39.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 113\n",
      "[362/723] Writing tensor blk.39.attn_norm.weight                | size   8192           | type F32  | T+ 114\n",
      "[363/723] Writing tensor blk.39.ffn_norm.weight                 | size   8192           | type F32  | T+ 114\n",
      "[364/723] Writing tensor blk.40.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 114\n",
      "[365/723] Writing tensor blk.40.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 114\n",
      "[366/723] Writing tensor blk.40.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 114\n",
      "[367/723] Writing tensor blk.40.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 114\n",
      "[368/723] Writing tensor blk.40.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 115\n",
      "[369/723] Writing tensor blk.40.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 116\n",
      "[370/723] Writing tensor blk.40.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 116\n",
      "[371/723] Writing tensor blk.40.attn_norm.weight                | size   8192           | type F32  | T+ 116\n",
      "[372/723] Writing tensor blk.40.ffn_norm.weight                 | size   8192           | type F32  | T+ 116\n",
      "[373/723] Writing tensor blk.41.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 116\n",
      "[374/723] Writing tensor blk.41.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 116\n",
      "[375/723] Writing tensor blk.41.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 116\n",
      "[376/723] Writing tensor blk.41.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 116\n",
      "[377/723] Writing tensor blk.41.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 118\n",
      "[378/723] Writing tensor blk.41.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 118\n",
      "[379/723] Writing tensor blk.41.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 118\n",
      "[380/723] Writing tensor blk.41.attn_norm.weight                | size   8192           | type F32  | T+ 119\n",
      "[381/723] Writing tensor blk.41.ffn_norm.weight                 | size   8192           | type F32  | T+ 119\n",
      "[382/723] Writing tensor blk.42.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 119\n",
      "[383/723] Writing tensor blk.42.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 119\n",
      "[384/723] Writing tensor blk.42.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 119\n",
      "[385/723] Writing tensor blk.42.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 119\n",
      "[386/723] Writing tensor blk.42.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 120\n",
      "[387/723] Writing tensor blk.42.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 121\n",
      "[388/723] Writing tensor blk.42.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 121\n",
      "[389/723] Writing tensor blk.42.attn_norm.weight                | size   8192           | type F32  | T+ 121\n",
      "[390/723] Writing tensor blk.42.ffn_norm.weight                 | size   8192           | type F32  | T+ 121\n",
      "[391/723] Writing tensor blk.43.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 121\n",
      "[392/723] Writing tensor blk.43.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 121\n",
      "[393/723] Writing tensor blk.43.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 121\n",
      "[394/723] Writing tensor blk.43.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 121\n",
      "[395/723] Writing tensor blk.43.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 123\n",
      "[396/723] Writing tensor blk.43.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 123\n",
      "[397/723] Writing tensor blk.43.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 123\n",
      "[398/723] Writing tensor blk.43.attn_norm.weight                | size   8192           | type F32  | T+ 124\n",
      "[399/723] Writing tensor blk.43.ffn_norm.weight                 | size   8192           | type F32  | T+ 124\n",
      "[400/723] Writing tensor blk.44.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 124\n",
      "[401/723] Writing tensor blk.44.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 124\n",
      "[402/723] Writing tensor blk.44.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 124\n",
      "[403/723] Writing tensor blk.44.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 124\n",
      "[404/723] Writing tensor blk.44.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 125\n",
      "[405/723] Writing tensor blk.44.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 126\n",
      "[406/723] Writing tensor blk.44.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 126\n",
      "[407/723] Writing tensor blk.44.attn_norm.weight                | size   8192           | type F32  | T+ 126\n",
      "[408/723] Writing tensor blk.44.ffn_norm.weight                 | size   8192           | type F32  | T+ 126\n",
      "[409/723] Writing tensor blk.45.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 126\n",
      "[410/723] Writing tensor blk.45.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 126\n",
      "[411/723] Writing tensor blk.45.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 126\n",
      "[412/723] Writing tensor blk.45.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 127\n",
      "[413/723] Writing tensor blk.45.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 130\n",
      "[414/723] Writing tensor blk.45.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 131\n",
      "[415/723] Writing tensor blk.45.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 131\n",
      "[416/723] Writing tensor blk.45.attn_norm.weight                | size   8192           | type F32  | T+ 131\n",
      "[417/723] Writing tensor blk.45.ffn_norm.weight                 | size   8192           | type F32  | T+ 131\n",
      "[418/723] Writing tensor blk.46.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 131\n",
      "[419/723] Writing tensor blk.46.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 131\n",
      "[420/723] Writing tensor blk.46.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 131\n",
      "[421/723] Writing tensor blk.46.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 131\n",
      "[422/723] Writing tensor blk.46.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 134\n",
      "[423/723] Writing tensor blk.46.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 134\n",
      "[424/723] Writing tensor blk.46.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 134\n",
      "[425/723] Writing tensor blk.46.attn_norm.weight                | size   8192           | type F32  | T+ 135\n",
      "[426/723] Writing tensor blk.46.ffn_norm.weight                 | size   8192           | type F32  | T+ 135\n",
      "[427/723] Writing tensor blk.47.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 135\n",
      "[428/723] Writing tensor blk.47.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 135\n",
      "[429/723] Writing tensor blk.47.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 135\n",
      "[430/723] Writing tensor blk.47.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 135\n",
      "[431/723] Writing tensor blk.47.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 136\n",
      "[432/723] Writing tensor blk.47.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 137\n",
      "[433/723] Writing tensor blk.47.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 137\n",
      "[434/723] Writing tensor blk.47.attn_norm.weight                | size   8192           | type F32  | T+ 137\n",
      "[435/723] Writing tensor blk.47.ffn_norm.weight                 | size   8192           | type F32  | T+ 137\n",
      "[436/723] Writing tensor blk.48.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 137\n",
      "[437/723] Writing tensor blk.48.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 137\n",
      "[438/723] Writing tensor blk.48.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 137\n",
      "[439/723] Writing tensor blk.48.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 137\n",
      "[440/723] Writing tensor blk.48.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 139\n",
      "[441/723] Writing tensor blk.48.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 139\n",
      "[442/723] Writing tensor blk.48.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 139\n",
      "[443/723] Writing tensor blk.48.attn_norm.weight                | size   8192           | type F32  | T+ 140\n",
      "[444/723] Writing tensor blk.48.ffn_norm.weight                 | size   8192           | type F32  | T+ 140\n",
      "[445/723] Writing tensor blk.49.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 140\n",
      "[446/723] Writing tensor blk.49.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 140\n",
      "[447/723] Writing tensor blk.49.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 140\n",
      "[448/723] Writing tensor blk.49.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 140\n",
      "[449/723] Writing tensor blk.49.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 141\n",
      "[450/723] Writing tensor blk.49.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 142\n",
      "[451/723] Writing tensor blk.49.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 142\n",
      "[452/723] Writing tensor blk.49.attn_norm.weight                | size   8192           | type F32  | T+ 142\n",
      "[453/723] Writing tensor blk.49.ffn_norm.weight                 | size   8192           | type F32  | T+ 142\n",
      "[454/723] Writing tensor blk.50.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 142\n",
      "[455/723] Writing tensor blk.50.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 142\n",
      "[456/723] Writing tensor blk.50.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 142\n",
      "[457/723] Writing tensor blk.50.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 142\n",
      "[458/723] Writing tensor blk.50.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 144\n",
      "[459/723] Writing tensor blk.50.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 144\n",
      "[460/723] Writing tensor blk.50.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 144\n",
      "[461/723] Writing tensor blk.50.attn_norm.weight                | size   8192           | type F32  | T+ 145\n",
      "[462/723] Writing tensor blk.50.ffn_norm.weight                 | size   8192           | type F32  | T+ 145\n",
      "[463/723] Writing tensor blk.51.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 145\n",
      "[464/723] Writing tensor blk.51.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 145\n",
      "[465/723] Writing tensor blk.51.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 145\n",
      "[466/723] Writing tensor blk.51.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 145\n",
      "[467/723] Writing tensor blk.51.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 146\n",
      "[468/723] Writing tensor blk.51.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 147\n",
      "[469/723] Writing tensor blk.51.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 147\n",
      "[470/723] Writing tensor blk.51.attn_norm.weight                | size   8192           | type F32  | T+ 147\n",
      "[471/723] Writing tensor blk.51.ffn_norm.weight                 | size   8192           | type F32  | T+ 147\n",
      "[472/723] Writing tensor blk.52.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 147\n",
      "[473/723] Writing tensor blk.52.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 147\n",
      "[474/723] Writing tensor blk.52.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 147\n",
      "[475/723] Writing tensor blk.52.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 147\n",
      "[476/723] Writing tensor blk.52.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 149\n",
      "[477/723] Writing tensor blk.52.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 149\n",
      "[478/723] Writing tensor blk.52.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 149\n",
      "[479/723] Writing tensor blk.52.attn_norm.weight                | size   8192           | type F32  | T+ 150\n",
      "[480/723] Writing tensor blk.52.ffn_norm.weight                 | size   8192           | type F32  | T+ 150\n",
      "[481/723] Writing tensor blk.53.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 150\n",
      "[482/723] Writing tensor blk.53.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 150\n",
      "[483/723] Writing tensor blk.53.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 150\n",
      "[484/723] Writing tensor blk.53.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 150\n",
      "[485/723] Writing tensor blk.53.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 151\n",
      "[486/723] Writing tensor blk.53.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 152\n",
      "[487/723] Writing tensor blk.53.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 152\n",
      "[488/723] Writing tensor blk.53.attn_norm.weight                | size   8192           | type F32  | T+ 152\n",
      "[489/723] Writing tensor blk.53.ffn_norm.weight                 | size   8192           | type F32  | T+ 152\n",
      "[490/723] Writing tensor blk.54.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 152\n",
      "[491/723] Writing tensor blk.54.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 152\n",
      "[492/723] Writing tensor blk.54.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 152\n",
      "[493/723] Writing tensor blk.54.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 152\n",
      "[494/723] Writing tensor blk.54.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 154\n",
      "[495/723] Writing tensor blk.54.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 154\n",
      "[496/723] Writing tensor blk.54.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 154\n",
      "[497/723] Writing tensor blk.54.attn_norm.weight                | size   8192           | type F32  | T+ 155\n",
      "[498/723] Writing tensor blk.54.ffn_norm.weight                 | size   8192           | type F32  | T+ 155\n",
      "[499/723] Writing tensor blk.55.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 155\n",
      "[500/723] Writing tensor blk.55.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 155\n",
      "[501/723] Writing tensor blk.55.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 155\n",
      "[502/723] Writing tensor blk.55.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 155\n",
      "[503/723] Writing tensor blk.55.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 156\n",
      "[504/723] Writing tensor blk.55.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 157\n",
      "[505/723] Writing tensor blk.55.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 157\n",
      "[506/723] Writing tensor blk.55.attn_norm.weight                | size   8192           | type F32  | T+ 157\n",
      "[507/723] Writing tensor blk.55.ffn_norm.weight                 | size   8192           | type F32  | T+ 157\n",
      "[508/723] Writing tensor blk.56.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 157\n",
      "[509/723] Writing tensor blk.56.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 157\n",
      "[510/723] Writing tensor blk.56.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 157\n",
      "[511/723] Writing tensor blk.56.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 157\n",
      "[512/723] Writing tensor blk.56.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 159\n",
      "[513/723] Writing tensor blk.56.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 159\n",
      "[514/723] Writing tensor blk.56.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 160\n",
      "[515/723] Writing tensor blk.56.attn_norm.weight                | size   8192           | type F32  | T+ 160\n",
      "[516/723] Writing tensor blk.56.ffn_norm.weight                 | size   8192           | type F32  | T+ 160\n",
      "[517/723] Writing tensor blk.57.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 160\n",
      "[518/723] Writing tensor blk.57.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 160\n",
      "[519/723] Writing tensor blk.57.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 160\n",
      "[520/723] Writing tensor blk.57.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 160\n",
      "[521/723] Writing tensor blk.57.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 164\n",
      "[522/723] Writing tensor blk.57.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 164\n",
      "[523/723] Writing tensor blk.57.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 164\n",
      "[524/723] Writing tensor blk.57.attn_norm.weight                | size   8192           | type F32  | T+ 165\n",
      "[525/723] Writing tensor blk.57.ffn_norm.weight                 | size   8192           | type F32  | T+ 165\n",
      "[526/723] Writing tensor blk.58.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 165\n",
      "[527/723] Writing tensor blk.58.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 165\n",
      "[528/723] Writing tensor blk.58.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 165\n",
      "[529/723] Writing tensor blk.58.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 165\n",
      "[530/723] Writing tensor blk.58.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 167\n",
      "[531/723] Writing tensor blk.58.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 167\n",
      "[532/723] Writing tensor blk.58.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 167\n",
      "[533/723] Writing tensor blk.58.attn_norm.weight                | size   8192           | type F32  | T+ 168\n",
      "[534/723] Writing tensor blk.58.ffn_norm.weight                 | size   8192           | type F32  | T+ 168\n",
      "[535/723] Writing tensor blk.59.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 168\n",
      "[536/723] Writing tensor blk.59.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 168\n",
      "[537/723] Writing tensor blk.59.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 168\n",
      "[538/723] Writing tensor blk.59.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 168\n",
      "[539/723] Writing tensor blk.59.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 170\n",
      "[540/723] Writing tensor blk.59.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 170\n",
      "[541/723] Writing tensor blk.59.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 170\n",
      "[542/723] Writing tensor blk.59.attn_norm.weight                | size   8192           | type F32  | T+ 170\n",
      "[543/723] Writing tensor blk.59.ffn_norm.weight                 | size   8192           | type F32  | T+ 170\n",
      "[544/723] Writing tensor blk.60.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 170\n",
      "[545/723] Writing tensor blk.60.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 170\n",
      "[546/723] Writing tensor blk.60.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 170\n",
      "[547/723] Writing tensor blk.60.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 170\n",
      "[548/723] Writing tensor blk.60.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 172\n",
      "[549/723] Writing tensor blk.60.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 172\n",
      "[550/723] Writing tensor blk.60.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 172\n",
      "[551/723] Writing tensor blk.60.attn_norm.weight                | size   8192           | type F32  | T+ 173\n",
      "[552/723] Writing tensor blk.60.ffn_norm.weight                 | size   8192           | type F32  | T+ 173\n",
      "[553/723] Writing tensor blk.61.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 173\n",
      "[554/723] Writing tensor blk.61.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 173\n",
      "[555/723] Writing tensor blk.61.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 173\n",
      "[556/723] Writing tensor blk.61.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 173\n",
      "[557/723] Writing tensor blk.61.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 174\n",
      "[558/723] Writing tensor blk.61.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 175\n",
      "[559/723] Writing tensor blk.61.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 175\n",
      "[560/723] Writing tensor blk.61.attn_norm.weight                | size   8192           | type F32  | T+ 175\n",
      "[561/723] Writing tensor blk.61.ffn_norm.weight                 | size   8192           | type F32  | T+ 175\n",
      "[562/723] Writing tensor blk.62.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 175\n",
      "[563/723] Writing tensor blk.62.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 175\n",
      "[564/723] Writing tensor blk.62.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 175\n",
      "[565/723] Writing tensor blk.62.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 175\n",
      "[566/723] Writing tensor blk.62.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 177\n",
      "[567/723] Writing tensor blk.62.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 177\n",
      "[568/723] Writing tensor blk.62.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 177\n",
      "[569/723] Writing tensor blk.62.attn_norm.weight                | size   8192           | type F32  | T+ 178\n",
      "[570/723] Writing tensor blk.62.ffn_norm.weight                 | size   8192           | type F32  | T+ 178\n",
      "[571/723] Writing tensor blk.63.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 178\n",
      "[572/723] Writing tensor blk.63.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 178\n",
      "[573/723] Writing tensor blk.63.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 178\n",
      "[574/723] Writing tensor blk.63.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 178\n",
      "[575/723] Writing tensor blk.63.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 180\n",
      "[576/723] Writing tensor blk.63.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 180\n",
      "[577/723] Writing tensor blk.63.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 180\n",
      "[578/723] Writing tensor blk.63.attn_norm.weight                | size   8192           | type F32  | T+ 180\n",
      "[579/723] Writing tensor blk.63.ffn_norm.weight                 | size   8192           | type F32  | T+ 180\n",
      "[580/723] Writing tensor blk.64.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 180\n",
      "[581/723] Writing tensor blk.64.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 180\n",
      "[582/723] Writing tensor blk.64.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 180\n",
      "[583/723] Writing tensor blk.64.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 180\n",
      "[584/723] Writing tensor blk.64.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 182\n",
      "[585/723] Writing tensor blk.64.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 182\n",
      "[586/723] Writing tensor blk.64.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 182\n",
      "[587/723] Writing tensor blk.64.attn_norm.weight                | size   8192           | type F32  | T+ 183\n",
      "[588/723] Writing tensor blk.64.ffn_norm.weight                 | size   8192           | type F32  | T+ 183\n",
      "[589/723] Writing tensor blk.65.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 183\n",
      "[590/723] Writing tensor blk.65.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 183\n",
      "[591/723] Writing tensor blk.65.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 183\n",
      "[592/723] Writing tensor blk.65.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 183\n",
      "[593/723] Writing tensor blk.65.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 184\n",
      "[594/723] Writing tensor blk.65.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 185\n",
      "[595/723] Writing tensor blk.65.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 185\n",
      "[596/723] Writing tensor blk.65.attn_norm.weight                | size   8192           | type F32  | T+ 185\n",
      "[597/723] Writing tensor blk.65.ffn_norm.weight                 | size   8192           | type F32  | T+ 185\n",
      "[598/723] Writing tensor blk.66.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 185\n",
      "[599/723] Writing tensor blk.66.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 185\n",
      "[600/723] Writing tensor blk.66.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 185\n",
      "[601/723] Writing tensor blk.66.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 185\n",
      "[602/723] Writing tensor blk.66.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 187\n",
      "[603/723] Writing tensor blk.66.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 187\n",
      "[604/723] Writing tensor blk.66.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 187\n",
      "[605/723] Writing tensor blk.66.attn_norm.weight                | size   8192           | type F32  | T+ 187\n",
      "[606/723] Writing tensor blk.66.ffn_norm.weight                 | size   8192           | type F32  | T+ 187\n",
      "[607/723] Writing tensor blk.67.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 187\n",
      "[608/723] Writing tensor blk.67.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 188\n",
      "[609/723] Writing tensor blk.67.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 188\n",
      "[610/723] Writing tensor blk.67.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 188\n",
      "[611/723] Writing tensor blk.67.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 189\n",
      "[612/723] Writing tensor blk.67.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 190\n",
      "[613/723] Writing tensor blk.67.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 190\n",
      "[614/723] Writing tensor blk.67.attn_norm.weight                | size   8192           | type F32  | T+ 190\n",
      "[615/723] Writing tensor blk.67.ffn_norm.weight                 | size   8192           | type F32  | T+ 190\n",
      "[616/723] Writing tensor blk.68.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 190\n",
      "[617/723] Writing tensor blk.68.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 190\n",
      "[618/723] Writing tensor blk.68.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 190\n",
      "[619/723] Writing tensor blk.68.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 191\n",
      "[620/723] Writing tensor blk.68.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 194\n",
      "[621/723] Writing tensor blk.68.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 194\n",
      "[622/723] Writing tensor blk.68.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 195\n",
      "[623/723] Writing tensor blk.68.attn_norm.weight                | size   8192           | type F32  | T+ 195\n",
      "[624/723] Writing tensor blk.68.ffn_norm.weight                 | size   8192           | type F32  | T+ 195\n",
      "[625/723] Writing tensor blk.69.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 195\n",
      "[626/723] Writing tensor blk.69.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 195\n",
      "[627/723] Writing tensor blk.69.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 195\n",
      "[628/723] Writing tensor blk.69.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 195\n",
      "[629/723] Writing tensor blk.69.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 197\n",
      "[630/723] Writing tensor blk.69.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 197\n",
      "[631/723] Writing tensor blk.69.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 197\n",
      "[632/723] Writing tensor blk.69.attn_norm.weight                | size   8192           | type F32  | T+ 197\n",
      "[633/723] Writing tensor blk.69.ffn_norm.weight                 | size   8192           | type F32  | T+ 197\n",
      "[634/723] Writing tensor blk.70.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 197\n",
      "[635/723] Writing tensor blk.70.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 197\n",
      "[636/723] Writing tensor blk.70.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 197\n",
      "[637/723] Writing tensor blk.70.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 197\n",
      "[638/723] Writing tensor blk.70.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 199\n",
      "[639/723] Writing tensor blk.70.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 200\n",
      "[640/723] Writing tensor blk.70.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 200\n",
      "[641/723] Writing tensor blk.70.attn_norm.weight                | size   8192           | type F32  | T+ 200\n",
      "[642/723] Writing tensor blk.70.ffn_norm.weight                 | size   8192           | type F32  | T+ 200\n",
      "[643/723] Writing tensor blk.71.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 200\n",
      "[644/723] Writing tensor blk.71.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 200\n",
      "[645/723] Writing tensor blk.71.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 200\n",
      "[646/723] Writing tensor blk.71.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 200\n",
      "[647/723] Writing tensor blk.71.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 202\n",
      "[648/723] Writing tensor blk.71.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 202\n",
      "[649/723] Writing tensor blk.71.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 202\n",
      "[650/723] Writing tensor blk.71.attn_norm.weight                | size   8192           | type F32  | T+ 202\n",
      "[651/723] Writing tensor blk.71.ffn_norm.weight                 | size   8192           | type F32  | T+ 202\n",
      "[652/723] Writing tensor blk.72.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 202\n",
      "[653/723] Writing tensor blk.72.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 202\n",
      "[654/723] Writing tensor blk.72.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 202\n",
      "[655/723] Writing tensor blk.72.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 202\n",
      "[656/723] Writing tensor blk.72.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 204\n",
      "[657/723] Writing tensor blk.72.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 204\n",
      "[658/723] Writing tensor blk.72.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 205\n",
      "[659/723] Writing tensor blk.72.attn_norm.weight                | size   8192           | type F32  | T+ 205\n",
      "[660/723] Writing tensor blk.72.ffn_norm.weight                 | size   8192           | type F32  | T+ 205\n",
      "[661/723] Writing tensor blk.73.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 205\n",
      "[662/723] Writing tensor blk.73.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 205\n",
      "[663/723] Writing tensor blk.73.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 205\n",
      "[664/723] Writing tensor blk.73.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 205\n",
      "[665/723] Writing tensor blk.73.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 207\n",
      "[666/723] Writing tensor blk.73.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 207\n",
      "[667/723] Writing tensor blk.73.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 207\n",
      "[668/723] Writing tensor blk.73.attn_norm.weight                | size   8192           | type F32  | T+ 207\n",
      "[669/723] Writing tensor blk.73.ffn_norm.weight                 | size   8192           | type F32  | T+ 207\n",
      "[670/723] Writing tensor blk.74.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 207\n",
      "[671/723] Writing tensor blk.74.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 207\n",
      "[672/723] Writing tensor blk.74.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 207\n",
      "[673/723] Writing tensor blk.74.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 207\n",
      "[674/723] Writing tensor blk.74.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 209\n",
      "[675/723] Writing tensor blk.74.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 209\n",
      "[676/723] Writing tensor blk.74.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 210\n",
      "[677/723] Writing tensor blk.74.attn_norm.weight                | size   8192           | type F32  | T+ 210\n",
      "[678/723] Writing tensor blk.74.ffn_norm.weight                 | size   8192           | type F32  | T+ 210\n",
      "[679/723] Writing tensor blk.75.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 210\n",
      "[680/723] Writing tensor blk.75.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 210\n",
      "[681/723] Writing tensor blk.75.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 210\n",
      "[682/723] Writing tensor blk.75.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 210\n",
      "[683/723] Writing tensor blk.75.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 212\n",
      "[684/723] Writing tensor blk.75.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 212\n",
      "[685/723] Writing tensor blk.75.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 212\n",
      "[686/723] Writing tensor blk.75.attn_norm.weight                | size   8192           | type F32  | T+ 212\n",
      "[687/723] Writing tensor blk.75.ffn_norm.weight                 | size   8192           | type F32  | T+ 212\n",
      "[688/723] Writing tensor blk.76.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 212\n",
      "[689/723] Writing tensor blk.76.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 212\n",
      "[690/723] Writing tensor blk.76.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 212\n",
      "[691/723] Writing tensor blk.76.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 212\n",
      "[692/723] Writing tensor blk.76.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 214\n",
      "[693/723] Writing tensor blk.76.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 214\n",
      "[694/723] Writing tensor blk.76.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 215\n",
      "[695/723] Writing tensor blk.76.attn_norm.weight                | size   8192           | type F32  | T+ 215\n",
      "[696/723] Writing tensor blk.76.ffn_norm.weight                 | size   8192           | type F32  | T+ 215\n",
      "[697/723] Writing tensor blk.77.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 215\n",
      "[698/723] Writing tensor blk.77.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 215\n",
      "[699/723] Writing tensor blk.77.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 215\n",
      "[700/723] Writing tensor blk.77.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 215\n",
      "[701/723] Writing tensor blk.77.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 217\n",
      "[702/723] Writing tensor blk.77.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 217\n",
      "[703/723] Writing tensor blk.77.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 217\n",
      "[704/723] Writing tensor blk.77.attn_norm.weight                | size   8192           | type F32  | T+ 217\n",
      "[705/723] Writing tensor blk.77.ffn_norm.weight                 | size   8192           | type F32  | T+ 217\n",
      "[706/723] Writing tensor blk.78.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 217\n",
      "[707/723] Writing tensor blk.78.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 218\n",
      "[708/723] Writing tensor blk.78.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 218\n",
      "[709/723] Writing tensor blk.78.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 218\n",
      "[710/723] Writing tensor blk.78.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 219\n",
      "[711/723] Writing tensor blk.78.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 220\n",
      "[712/723] Writing tensor blk.78.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 220\n",
      "[713/723] Writing tensor blk.78.attn_norm.weight                | size   8192           | type F32  | T+ 220\n",
      "[714/723] Writing tensor blk.78.ffn_norm.weight                 | size   8192           | type F32  | T+ 220\n",
      "[715/723] Writing tensor blk.79.attn_q.weight                   | size   8192 x   8192  | type F16  | T+ 220\n",
      "[716/723] Writing tensor blk.79.attn_k.weight                   | size   1024 x   8192  | type F16  | T+ 220\n",
      "[717/723] Writing tensor blk.79.attn_v.weight                   | size   1024 x   8192  | type F16  | T+ 220\n",
      "[718/723] Writing tensor blk.79.attn_output.weight              | size   8192 x   8192  | type F16  | T+ 221\n",
      "[719/723] Writing tensor blk.79.ffn_gate.weight                 | size  28672 x   8192  | type F16  | T+ 224\n",
      "[720/723] Writing tensor blk.79.ffn_down.weight                 | size   8192 x  28672  | type F16  | T+ 224\n",
      "[721/723] Writing tensor blk.79.ffn_up.weight                   | size  28672 x   8192  | type F16  | T+ 225\n",
      "[722/723] Writing tensor blk.79.attn_norm.weight                | size   8192           | type F32  | T+ 225\n",
      "[723/723] Writing tensor blk.79.ffn_norm.weight                 | size   8192           | type F32  | T+ 225\n",
      "Wrote models/70B-v2/ggml-model-f16.gguf\n"
     ]
    }
   ],
   "source": [
    "# convert the models to ggml FP16 format\n",
    "!python3 convert.py models/7B-v2/\n",
    "!python3 convert.py models/13B-v2/\n",
    "!python3 convert.py models/70B-v2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "954d1eb9-d1d6-4525-8b0f-3b5809ad2d84",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I llama.cpp build info: \n",
      "I UNAME_S:   Linux\n",
      "I UNAME_P:   x86_64\n",
      "I UNAME_M:   x86_64\n",
      "I CFLAGS:    -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion \n",
      "I CXXFLAGS:  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi\n",
      "I NVCCFLAGS:  \n",
      "I LDFLAGS:    \n",
      "I CC:        cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "I CXX:       g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "\n",
      "rm -vrf *.o tests/*.o *.so *.dll benchmark-matmult common/build-info.cpp *.dot *.gcno tests/*.gcno *.gcda tests/*.gcda *.gcov tests/*.gcov lcov-report gcovr-report main quantize quantize-stats perplexity embedding vdot q8dot train-text-from-scratch convert-llama2c-to-ggml simple batched batched-bench save-load-state server gguf llama-bench libllava.a llava-cli baby-llama beam-search speculative infill tokenize benchmark-matmult parallel finetune export-lora lookahead lookup tests/test-c.o metal tests/test-llama-grammar tests/test-grammar-parser tests/test-double-float tests/test-grad0 tests/test-opt tests/test-quantize-fns tests/test-quantize-perf tests/test-sampling tests/test-tokenizer-0-llama tests/test-tokenizer-0-falcon tests/test-tokenizer-1-llama tests/test-tokenizer-1-bpe tests/test-rope tests/test-backend-ops\n",
      "I llama.cpp build info: \n",
      "I UNAME_S:   Linux\n",
      "I UNAME_P:   x86_64\n",
      "I UNAME_M:   x86_64\n",
      "I CFLAGS:    -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion \n",
      "I CXXFLAGS:  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi\n",
      "I NVCCFLAGS: -use_fast_math --forward-unknown-to-host-compiler -arch=native -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 \n",
      "I LDFLAGS:   -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "I CC:        cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "I CXX:       g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "\n",
      "cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml.c -o ggml.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c llama.cpp -o llama.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/common.cpp -o common.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/sampling.cpp -o sampling.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/grammar-parser.cpp -o grammar-parser.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/console.cpp -o console.o\n",
      "nvcc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -use_fast_math --forward-unknown-to-host-compiler -arch=native -DGGML_CUDA_DMMV_X=32 -DGGML_CUDA_MMV_Y=1 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128  -Wno-pedantic -Xcompiler \"-Wno-array-bounds -Wno-format-truncation -Wextra-semi\" -c ggml-cuda.cu -o ggml-cuda.o\n",
      "cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml-alloc.c -o ggml-alloc.o\n",
      "cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml-backend.c -o ggml-backend.o\n",
      "cc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion     -c ggml-quants.c -o ggml-quants.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/train.cpp -o train.o\n",
      "cc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion  -c tests/test-c.c -o tests/test-c.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -c common/build-info.cpp -o build-info.o\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi pocs/vdot/vdot.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o vdot -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi pocs/vdot/q8dot.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o q8dot -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/gguf/gguf.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o gguf -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/benchmark/benchmark-matmult.cpp build-info.o ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o benchmark-matmult -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/export-lora/export-lora.cpp ggml.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o export-lora -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/main/main.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o console.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o main -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/quantize/quantize.cpp build-info.o ggml.o llama.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o quantize -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/quantize-stats/quantize-stats.cpp build-info.o ggml.o llama.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o quantize-stats -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/perplexity/perplexity.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o perplexity -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/embedding/embedding.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o embedding -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/train-text-from-scratch/train-text-from-scratch.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o train.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o train-text-from-scratch -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/convert-llama2c-to-ggml/convert-llama2c-to-ggml.cpp ggml.o llama.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o convert-llama2c-to-ggml -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/simple/simple.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o simple -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/batched/batched.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o batched -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/batched-bench/batched-bench.cpp build-info.o ggml.o llama.o common.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o batched-bench -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/save-load-state/save-load-state.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o save-load-state -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -Iexamples/server examples/server/server.cpp examples/llava/clip.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o server -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib   -Wno-cast-qual\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/llama-bench/llama-bench.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o llama-bench -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -static -fPIC -c examples/llava/llava.cpp -o libllava.a -Wno-cast-qual\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/llava/llava-cli.cpp examples/llava/clip.cpp examples/llava/llava.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o llava-cli -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib  -Wno-cast-qual\n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/baby-llama/baby-llama.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o train.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o baby-llama -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/beam-search/beam-search.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o beam-search -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/speculative/speculative.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o speculative -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/infill/infill.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o console.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o infill -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/tokenize/tokenize.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o tokenize -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/parallel/parallel.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o parallel -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/finetune/finetune.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o train.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o finetune -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/lookahead/lookahead.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o lookahead -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "g++ -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG -DGGML_USE_CUBLAS -I/usr/local/cuda/include -I/opt/cuda/include -I/targets/x86_64-linux/include -I/usr/local/cuda/targets/aarch64-linux/include  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi examples/lookup/lookup.cpp ggml.o llama.o common.o sampling.o grammar-parser.o build-info.o ggml-cuda.o ggml-alloc.o ggml-backend.o ggml-quants.o -o lookup -lcublas -lculibos -lcudart -lcublasLt -lpthread -ldl -lrt -L/usr/local/cuda/lib64 -L/opt/cuda/lib64 -L/targets/x86_64-linux/lib -L/usr/local/cuda/targets/aarch64-linux/lib \n",
      "\n",
      "====  Run ./main -h for help.  ====\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# metal build\n",
    "!make clean && LLAMA_CUBLAS=1 make -j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c99bdabe-ce05-4e4a-bb7f-1ad00b66e57e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: quantizing './models/7B-v2/ggml-model-f16.gguf' to './models/7B-v2/ggml-model-q4_0.gguf' as Q4_0\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llama_model_quantize_internal: meta size = 1714336 bytes\n",
      "[   1/ 291]                    token_embd.weight - [ 4096, 32000,     1,     1], type =    f16, quantizing to q4_0 .. size =   250.00 MiB ->    70.31 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   2/ 291]                   output_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[   3/ 291]                        output.weight - [ 4096, 32000,     1,     1], type =    f16, quantizing to q6_K .. size =   250.00 MiB ->   102.54 MiB | hist: \n",
      "[   4/ 291]                  blk.0.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.034 0.008 0.012 0.019 0.031 0.050 0.084 0.149 0.256 0.150 0.084 0.051 0.031 0.019 0.012 0.010 \n",
      "[   5/ 291]                  blk.0.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.034 0.008 0.013 0.021 0.033 0.054 0.089 0.150 0.226 0.151 0.089 0.054 0.033 0.021 0.013 0.011 \n",
      "[   6/ 291]                  blk.0.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.024 0.036 0.053 0.074 0.096 0.117 0.129 0.117 0.096 0.074 0.053 0.036 0.024 0.020 \n",
      "[   7/ 291]             blk.0.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.035 0.011 0.017 0.028 0.044 0.068 0.100 0.135 0.155 0.135 0.100 0.068 0.044 0.028 0.017 0.014 \n",
      "[   8/ 291]                blk.0.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   9/ 291]                blk.0.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  10/ 291]                  blk.0.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  11/ 291]               blk.0.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  12/ 291]                blk.0.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  13/ 291]                  blk.1.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.013 0.022 0.034 0.052 0.074 0.098 0.121 0.132 0.121 0.098 0.074 0.052 0.034 0.022 0.018 \n",
      "[  14/ 291]                  blk.1.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.013 0.022 0.034 0.051 0.074 0.099 0.121 0.132 0.121 0.099 0.074 0.051 0.034 0.022 0.018 \n",
      "[  15/ 291]                  blk.1.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.014 0.023 0.035 0.052 0.073 0.097 0.119 0.130 0.119 0.097 0.074 0.052 0.035 0.023 0.019 \n",
      "[  16/ 291]             blk.1.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.035 0.012 0.020 0.031 0.047 0.070 0.098 0.129 0.146 0.129 0.099 0.070 0.047 0.031 0.020 0.016 \n",
      "[  17/ 291]                blk.1.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  18/ 291]                blk.1.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  19/ 291]                  blk.1.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  20/ 291]               blk.1.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  21/ 291]                blk.1.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  22/ 291]                  blk.2.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.096 0.114 0.122 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  23/ 291]                  blk.2.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.124 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[  24/ 291]                  blk.2.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.120 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  25/ 291]             blk.2.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  26/ 291]                blk.2.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  27/ 291]                blk.2.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  28/ 291]                  blk.2.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  29/ 291]               blk.2.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  30/ 291]                blk.2.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  31/ 291]                  blk.3.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  32/ 291]                  blk.3.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  33/ 291]                  blk.3.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  34/ 291]             blk.3.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  35/ 291]                blk.3.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  36/ 291]                blk.3.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  37/ 291]                  blk.3.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  38/ 291]               blk.3.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  39/ 291]                blk.3.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  40/ 291]                  blk.4.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[  41/ 291]                  blk.4.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  42/ 291]                  blk.4.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  43/ 291]             blk.4.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  44/ 291]                blk.4.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  45/ 291]                blk.4.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  46/ 291]                  blk.4.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  47/ 291]               blk.4.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  48/ 291]                blk.4.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  49/ 291]                  blk.5.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  50/ 291]                  blk.5.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  51/ 291]                  blk.5.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  52/ 291]             blk.5.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  53/ 291]                blk.5.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  54/ 291]                blk.5.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  55/ 291]                  blk.5.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  56/ 291]               blk.5.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  57/ 291]                blk.5.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  58/ 291]                  blk.6.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  59/ 291]                  blk.6.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  60/ 291]                  blk.6.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  61/ 291]             blk.6.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  62/ 291]                blk.6.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  63/ 291]                blk.6.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  64/ 291]                  blk.6.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  65/ 291]               blk.6.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  66/ 291]                blk.6.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  67/ 291]                  blk.7.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  68/ 291]                  blk.7.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  69/ 291]                  blk.7.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  70/ 291]             blk.7.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  71/ 291]                blk.7.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  72/ 291]                blk.7.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  73/ 291]                  blk.7.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  74/ 291]               blk.7.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  75/ 291]                blk.7.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  76/ 291]                  blk.8.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  77/ 291]                  blk.8.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  78/ 291]                  blk.8.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  79/ 291]             blk.8.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  80/ 291]                blk.8.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  81/ 291]                blk.8.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  82/ 291]                  blk.8.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  83/ 291]               blk.8.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  84/ 291]                blk.8.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  85/ 291]                  blk.9.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  86/ 291]                  blk.9.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  87/ 291]                  blk.9.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  88/ 291]             blk.9.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  89/ 291]                blk.9.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  90/ 291]                blk.9.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  91/ 291]                  blk.9.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  92/ 291]               blk.9.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  93/ 291]                blk.9.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[  94/ 291]                 blk.10.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  95/ 291]                 blk.10.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  96/ 291]                 blk.10.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  97/ 291]            blk.10.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  98/ 291]               blk.10.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  99/ 291]               blk.10.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 100/ 291]                 blk.10.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 101/ 291]              blk.10.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 102/ 291]               blk.10.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 103/ 291]                 blk.11.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 104/ 291]                 blk.11.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 105/ 291]                 blk.11.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 106/ 291]            blk.11.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 107/ 291]               blk.11.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 108/ 291]               blk.11.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 109/ 291]                 blk.11.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 110/ 291]              blk.11.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 111/ 291]               blk.11.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 112/ 291]                 blk.12.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 113/ 291]                 blk.12.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 114/ 291]                 blk.12.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 115/ 291]            blk.12.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 116/ 291]               blk.12.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 117/ 291]               blk.12.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 118/ 291]                 blk.12.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 119/ 291]              blk.12.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 120/ 291]               blk.12.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 121/ 291]                 blk.13.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 122/ 291]                 blk.13.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 123/ 291]                 blk.13.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 124/ 291]            blk.13.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 125/ 291]               blk.13.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 126/ 291]               blk.13.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 127/ 291]                 blk.13.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 128/ 291]              blk.13.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 129/ 291]               blk.13.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 130/ 291]                 blk.14.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 131/ 291]                 blk.14.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 132/ 291]                 blk.14.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 133/ 291]            blk.14.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 134/ 291]               blk.14.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 135/ 291]               blk.14.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 136/ 291]                 blk.14.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 137/ 291]              blk.14.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 138/ 291]               blk.14.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 139/ 291]                 blk.15.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 140/ 291]                 blk.15.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 141/ 291]                 blk.15.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 142/ 291]            blk.15.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 143/ 291]               blk.15.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 144/ 291]               blk.15.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 145/ 291]                 blk.15.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 146/ 291]              blk.15.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 147/ 291]               blk.15.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 148/ 291]                 blk.16.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 149/ 291]                 blk.16.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 150/ 291]                 blk.16.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 151/ 291]            blk.16.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 152/ 291]               blk.16.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 153/ 291]               blk.16.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 154/ 291]                 blk.16.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 155/ 291]              blk.16.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 156/ 291]               blk.16.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 157/ 291]                 blk.17.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 158/ 291]                 blk.17.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 159/ 291]                 blk.17.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 160/ 291]            blk.17.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 161/ 291]               blk.17.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 162/ 291]               blk.17.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 163/ 291]                 blk.17.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 164/ 291]              blk.17.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 165/ 291]               blk.17.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 166/ 291]                 blk.18.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 167/ 291]                 blk.18.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 168/ 291]                 blk.18.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 169/ 291]            blk.18.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 170/ 291]               blk.18.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 171/ 291]               blk.18.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 172/ 291]                 blk.18.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 173/ 291]              blk.18.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 174/ 291]               blk.18.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 175/ 291]                 blk.19.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 176/ 291]                 blk.19.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 177/ 291]                 blk.19.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 178/ 291]            blk.19.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 179/ 291]               blk.19.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 180/ 291]               blk.19.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 181/ 291]                 blk.19.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 182/ 291]              blk.19.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 183/ 291]               blk.19.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 184/ 291]                 blk.20.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 185/ 291]                 blk.20.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 186/ 291]                 blk.20.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 187/ 291]            blk.20.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 188/ 291]               blk.20.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 189/ 291]               blk.20.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 190/ 291]                 blk.20.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 191/ 291]              blk.20.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 192/ 291]               blk.20.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 193/ 291]                 blk.21.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 194/ 291]                 blk.21.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 195/ 291]                 blk.21.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 196/ 291]            blk.21.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 197/ 291]               blk.21.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 198/ 291]               blk.21.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 199/ 291]                 blk.21.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 200/ 291]              blk.21.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 201/ 291]               blk.21.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 202/ 291]                 blk.22.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 203/ 291]                 blk.22.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 204/ 291]                 blk.22.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 205/ 291]            blk.22.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 206/ 291]               blk.22.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 207/ 291]               blk.22.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 208/ 291]                 blk.22.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 209/ 291]              blk.22.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 210/ 291]               blk.22.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 211/ 291]                 blk.23.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 212/ 291]                 blk.23.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 213/ 291]                 blk.23.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 214/ 291]            blk.23.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 215/ 291]               blk.23.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 216/ 291]               blk.23.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 217/ 291]                 blk.23.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 218/ 291]              blk.23.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 219/ 291]               blk.23.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 220/ 291]                 blk.24.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 221/ 291]                 blk.24.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 222/ 291]                 blk.24.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 223/ 291]            blk.24.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 224/ 291]               blk.24.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 225/ 291]               blk.24.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 226/ 291]                 blk.24.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 227/ 291]              blk.24.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 228/ 291]               blk.24.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 229/ 291]                 blk.25.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 230/ 291]                 blk.25.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 231/ 291]                 blk.25.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 232/ 291]            blk.25.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 233/ 291]               blk.25.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 234/ 291]               blk.25.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 235/ 291]                 blk.25.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 236/ 291]              blk.25.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 237/ 291]               blk.25.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 238/ 291]                 blk.26.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 239/ 291]                 blk.26.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 240/ 291]                 blk.26.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 241/ 291]            blk.26.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 242/ 291]               blk.26.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 243/ 291]               blk.26.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 244/ 291]                 blk.26.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 245/ 291]              blk.26.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 246/ 291]               blk.26.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 247/ 291]                 blk.27.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 248/ 291]                 blk.27.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 249/ 291]                 blk.27.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 250/ 291]            blk.27.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 251/ 291]               blk.27.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 252/ 291]               blk.27.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 253/ 291]                 blk.27.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 254/ 291]              blk.27.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 255/ 291]               blk.27.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 256/ 291]                 blk.28.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 257/ 291]                 blk.28.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 258/ 291]                 blk.28.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 259/ 291]            blk.28.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 260/ 291]               blk.28.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 261/ 291]               blk.28.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 262/ 291]                 blk.28.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 263/ 291]              blk.28.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 264/ 291]               blk.28.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 265/ 291]                 blk.29.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 266/ 291]                 blk.29.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 267/ 291]                 blk.29.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 268/ 291]            blk.29.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 269/ 291]               blk.29.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 270/ 291]               blk.29.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 271/ 291]                 blk.29.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 272/ 291]              blk.29.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 273/ 291]               blk.29.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 274/ 291]                 blk.30.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 275/ 291]                 blk.30.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 276/ 291]                 blk.30.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 277/ 291]            blk.30.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 278/ 291]               blk.30.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 279/ 291]               blk.30.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.120 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 280/ 291]                 blk.30.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 281/ 291]              blk.30.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 282/ 291]               blk.30.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 283/ 291]                 blk.31.attn_q.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 284/ 291]                 blk.31.attn_k.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 285/ 291]                 blk.31.attn_v.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 286/ 291]            blk.31.attn_output.weight - [ 4096,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    32.00 MiB ->     9.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 287/ 291]               blk.31.ffn_gate.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 288/ 291]               blk.31.ffn_down.weight - [11008,  4096,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.015 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.116 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[ 289/ 291]                 blk.31.ffn_up.weight - [ 4096, 11008,     1,     1], type =    f16, quantizing to q4_0 .. size =    86.00 MiB ->    24.19 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 290/ 291]              blk.31.attn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "[ 291/ 291]               blk.31.ffn_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB\n",
      "llama_model_quantize_internal: model size  = 12853.02 MB\n",
      "llama_model_quantize_internal: quant size  =  3647.87 MB\n",
      "llama_model_quantize_internal: hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "\n",
      "main: quantize time = 14476.15 ms\n",
      "main:    total time = 14476.15 ms\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: quantizing './models/13B-v2/ggml-model-f16.gguf' to './models/13B-v2/ggml-model-q4_0.gguf' as Q4_0\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type  f16:  282 tensors\n",
      "llama_model_quantize_internal: meta size = 1718656 bytes\n",
      "[   1/ 363]                    token_embd.weight - [ 5120, 32000,     1,     1], type =    f16, quantizing to q4_0 .. size =   312.50 MiB ->    87.89 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   2/ 363]                   output_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[   3/ 363]                        output.weight - [ 5120, 32000,     1,     1], type =    f16, quantizing to q6_K .. size =   312.50 MiB ->   128.17 MiB | hist: \n",
      "[   4/ 363]                  blk.0.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.033 0.006 0.009 0.015 0.024 0.041 0.074 0.153 0.317 0.153 0.075 0.041 0.024 0.015 0.009 0.008 \n",
      "[   5/ 363]                  blk.0.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.033 0.006 0.010 0.015 0.025 0.043 0.078 0.158 0.293 0.158 0.078 0.043 0.025 0.015 0.010 0.008 \n",
      "[   6/ 363]                  blk.0.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.014 0.023 0.035 0.053 0.074 0.097 0.118 0.129 0.119 0.098 0.074 0.053 0.035 0.023 0.019 \n",
      "[   7/ 363]             blk.0.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.035 0.012 0.020 0.031 0.048 0.071 0.099 0.127 0.142 0.127 0.099 0.071 0.048 0.031 0.020 0.016 \n",
      "[   8/ 363]                blk.0.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[   9/ 363]                blk.0.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  10/ 363]                  blk.0.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  11/ 363]               blk.0.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  12/ 363]                blk.0.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  13/ 363]                  blk.1.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.013 0.021 0.033 0.050 0.072 0.098 0.124 0.139 0.124 0.098 0.072 0.050 0.033 0.021 0.017 \n",
      "[  14/ 363]                  blk.1.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.013 0.020 0.032 0.049 0.072 0.099 0.125 0.139 0.126 0.099 0.072 0.049 0.032 0.020 0.017 \n",
      "[  15/ 363]                  blk.1.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.054 0.075 0.097 0.116 0.124 0.116 0.097 0.075 0.054 0.037 0.024 0.020 \n",
      "[  16/ 363]             blk.1.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.013 0.021 0.033 0.051 0.073 0.099 0.123 0.134 0.123 0.099 0.073 0.051 0.034 0.021 0.018 \n",
      "[  17/ 363]                blk.1.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  18/ 363]                blk.1.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  19/ 363]                  blk.1.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  20/ 363]               blk.1.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  21/ 363]                blk.1.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  22/ 363]                  blk.2.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.123 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  23/ 363]                  blk.2.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.124 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[  24/ 363]                  blk.2.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  25/ 363]             blk.2.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.024 0.020 \n",
      "[  26/ 363]                blk.2.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  27/ 363]                blk.2.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  28/ 363]                  blk.2.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  29/ 363]               blk.2.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  30/ 363]                blk.2.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  31/ 363]                  blk.3.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.096 0.113 0.121 0.113 0.097 0.076 0.055 0.038 0.025 0.020 \n",
      "[  32/ 363]                  blk.3.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[  33/ 363]                  blk.3.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  34/ 363]             blk.3.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  35/ 363]                blk.3.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  36/ 363]                blk.3.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  37/ 363]                  blk.3.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  38/ 363]               blk.3.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  39/ 363]                blk.3.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  40/ 363]                  blk.4.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  41/ 363]                  blk.4.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.113 0.120 0.113 0.096 0.076 0.056 0.038 0.025 0.020 \n",
      "[  42/ 363]                  blk.4.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[  43/ 363]             blk.4.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  44/ 363]                blk.4.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  45/ 363]                blk.4.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  46/ 363]                  blk.4.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  47/ 363]               blk.4.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  48/ 363]                blk.4.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  49/ 363]                  blk.5.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  50/ 363]                  blk.5.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  51/ 363]                  blk.5.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  52/ 363]             blk.5.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  53/ 363]                blk.5.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  54/ 363]                blk.5.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  55/ 363]                  blk.5.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  56/ 363]               blk.5.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  57/ 363]                blk.5.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  58/ 363]                  blk.6.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  59/ 363]                  blk.6.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  60/ 363]                  blk.6.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  61/ 363]             blk.6.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  62/ 363]                blk.6.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  63/ 363]                blk.6.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  64/ 363]                  blk.6.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  65/ 363]               blk.6.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  66/ 363]                blk.6.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  67/ 363]                  blk.7.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  68/ 363]                  blk.7.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  69/ 363]                  blk.7.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  70/ 363]             blk.7.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  71/ 363]                blk.7.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  72/ 363]                blk.7.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  73/ 363]                  blk.7.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  74/ 363]               blk.7.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  75/ 363]                blk.7.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  76/ 363]                  blk.8.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  77/ 363]                  blk.8.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  78/ 363]                  blk.8.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  79/ 363]             blk.8.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  80/ 363]                blk.8.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  81/ 363]                blk.8.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  82/ 363]                  blk.8.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  83/ 363]               blk.8.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  84/ 363]                blk.8.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  85/ 363]                  blk.9.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  86/ 363]                  blk.9.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  87/ 363]                  blk.9.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  88/ 363]             blk.9.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  89/ 363]                blk.9.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  90/ 363]                blk.9.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  91/ 363]                  blk.9.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  92/ 363]               blk.9.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  93/ 363]                blk.9.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[  94/ 363]                 blk.10.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  95/ 363]                 blk.10.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  96/ 363]                 blk.10.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[  97/ 363]            blk.10.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  98/ 363]               blk.10.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  99/ 363]               blk.10.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 100/ 363]                 blk.10.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 101/ 363]              blk.10.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 102/ 363]               blk.10.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 103/ 363]                 blk.11.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 104/ 363]                 blk.11.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 105/ 363]                 blk.11.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 106/ 363]            blk.11.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 107/ 363]               blk.11.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 108/ 363]               blk.11.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 109/ 363]                 blk.11.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 110/ 363]              blk.11.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 111/ 363]               blk.11.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 112/ 363]                 blk.12.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 113/ 363]                 blk.12.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 114/ 363]                 blk.12.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 115/ 363]            blk.12.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 116/ 363]               blk.12.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 117/ 363]               blk.12.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 118/ 363]                 blk.12.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 119/ 363]              blk.12.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 120/ 363]               blk.12.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 121/ 363]                 blk.13.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 122/ 363]                 blk.13.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 123/ 363]                 blk.13.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 124/ 363]            blk.13.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 125/ 363]               blk.13.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 126/ 363]               blk.13.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 127/ 363]                 blk.13.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 128/ 363]              blk.13.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 129/ 363]               blk.13.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 130/ 363]                 blk.14.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 131/ 363]                 blk.14.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 132/ 363]                 blk.14.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 133/ 363]            blk.14.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 134/ 363]               blk.14.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 135/ 363]               blk.14.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 136/ 363]                 blk.14.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 137/ 363]              blk.14.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 138/ 363]               blk.14.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 139/ 363]                 blk.15.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 140/ 363]                 blk.15.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 141/ 363]                 blk.15.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 142/ 363]            blk.15.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 143/ 363]               blk.15.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 144/ 363]               blk.15.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 145/ 363]                 blk.15.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 146/ 363]              blk.15.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 147/ 363]               blk.15.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 148/ 363]                 blk.16.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 149/ 363]                 blk.16.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 150/ 363]                 blk.16.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 151/ 363]            blk.16.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 152/ 363]               blk.16.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 153/ 363]               blk.16.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 154/ 363]                 blk.16.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 155/ 363]              blk.16.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 156/ 363]               blk.16.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 157/ 363]                 blk.17.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 158/ 363]                 blk.17.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 159/ 363]                 blk.17.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 160/ 363]            blk.17.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 161/ 363]               blk.17.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 162/ 363]               blk.17.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 163/ 363]                 blk.17.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 164/ 363]              blk.17.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 165/ 363]               blk.17.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 166/ 363]                 blk.18.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 167/ 363]                 blk.18.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 168/ 363]                 blk.18.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 169/ 363]            blk.18.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 170/ 363]               blk.18.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 171/ 363]               blk.18.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 172/ 363]                 blk.18.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 173/ 363]              blk.18.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 174/ 363]               blk.18.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 175/ 363]                 blk.19.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 176/ 363]                 blk.19.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 177/ 363]                 blk.19.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 178/ 363]            blk.19.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 179/ 363]               blk.19.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 180/ 363]               blk.19.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 181/ 363]                 blk.19.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 182/ 363]              blk.19.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 183/ 363]               blk.19.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 184/ 363]                 blk.20.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 185/ 363]                 blk.20.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 186/ 363]                 blk.20.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 187/ 363]            blk.20.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 188/ 363]               blk.20.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 189/ 363]               blk.20.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 190/ 363]                 blk.20.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 191/ 363]              blk.20.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 192/ 363]               blk.20.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 193/ 363]                 blk.21.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 194/ 363]                 blk.21.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 195/ 363]                 blk.21.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 196/ 363]            blk.21.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 197/ 363]               blk.21.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 198/ 363]               blk.21.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 199/ 363]                 blk.21.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 200/ 363]              blk.21.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 201/ 363]               blk.21.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 202/ 363]                 blk.22.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 203/ 363]                 blk.22.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 204/ 363]                 blk.22.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 205/ 363]            blk.22.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 206/ 363]               blk.22.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 207/ 363]               blk.22.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 208/ 363]                 blk.22.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 209/ 363]              blk.22.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 210/ 363]               blk.22.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 211/ 363]                 blk.23.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 212/ 363]                 blk.23.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 213/ 363]                 blk.23.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 214/ 363]            blk.23.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 215/ 363]               blk.23.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 216/ 363]               blk.23.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 217/ 363]                 blk.23.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 218/ 363]              blk.23.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 219/ 363]               blk.23.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 220/ 363]                 blk.24.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 221/ 363]                 blk.24.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 222/ 363]                 blk.24.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 223/ 363]            blk.24.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 224/ 363]               blk.24.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 225/ 363]               blk.24.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 226/ 363]                 blk.24.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 227/ 363]              blk.24.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 228/ 363]               blk.24.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 229/ 363]                 blk.25.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 230/ 363]                 blk.25.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 231/ 363]                 blk.25.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 232/ 363]            blk.25.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 233/ 363]               blk.25.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 234/ 363]               blk.25.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 235/ 363]                 blk.25.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 236/ 363]              blk.25.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 237/ 363]               blk.25.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 238/ 363]                 blk.26.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 239/ 363]                 blk.26.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 240/ 363]                 blk.26.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 241/ 363]            blk.26.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 242/ 363]               blk.26.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 243/ 363]               blk.26.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 244/ 363]                 blk.26.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 245/ 363]              blk.26.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 246/ 363]               blk.26.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 247/ 363]                 blk.27.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 248/ 363]                 blk.27.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 249/ 363]                 blk.27.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 250/ 363]            blk.27.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 251/ 363]               blk.27.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 252/ 363]               blk.27.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 253/ 363]                 blk.27.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 254/ 363]              blk.27.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 255/ 363]               blk.27.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 256/ 363]                 blk.28.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 257/ 363]                 blk.28.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 258/ 363]                 blk.28.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 259/ 363]            blk.28.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 260/ 363]               blk.28.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 261/ 363]               blk.28.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 262/ 363]                 blk.28.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 263/ 363]              blk.28.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 264/ 363]               blk.28.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 265/ 363]                 blk.29.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 266/ 363]                 blk.29.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 267/ 363]                 blk.29.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 268/ 363]            blk.29.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 269/ 363]               blk.29.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 270/ 363]               blk.29.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 271/ 363]                 blk.29.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 272/ 363]              blk.29.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 273/ 363]               blk.29.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 274/ 363]                 blk.30.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 275/ 363]                 blk.30.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 276/ 363]                 blk.30.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 277/ 363]            blk.30.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 278/ 363]               blk.30.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 279/ 363]               blk.30.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 280/ 363]                 blk.30.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 281/ 363]              blk.30.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 282/ 363]               blk.30.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 283/ 363]                 blk.31.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 284/ 363]                 blk.31.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 285/ 363]                 blk.31.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 286/ 363]            blk.31.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 287/ 363]               blk.31.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 288/ 363]               blk.31.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 289/ 363]                 blk.31.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 290/ 363]              blk.31.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 291/ 363]               blk.31.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 292/ 363]                 blk.32.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 293/ 363]                 blk.32.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 294/ 363]                 blk.32.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 295/ 363]            blk.32.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 296/ 363]               blk.32.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 297/ 363]               blk.32.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 298/ 363]                 blk.32.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 299/ 363]              blk.32.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 300/ 363]               blk.32.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 301/ 363]                 blk.33.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 302/ 363]                 blk.33.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 303/ 363]                 blk.33.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 304/ 363]            blk.33.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 305/ 363]               blk.33.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 306/ 363]               blk.33.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 307/ 363]                 blk.33.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 308/ 363]              blk.33.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 309/ 363]               blk.33.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 310/ 363]                 blk.34.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 311/ 363]                 blk.34.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 312/ 363]                 blk.34.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 313/ 363]            blk.34.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 314/ 363]               blk.34.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 315/ 363]               blk.34.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 316/ 363]                 blk.34.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 317/ 363]              blk.34.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 318/ 363]               blk.34.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 319/ 363]                 blk.35.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 320/ 363]                 blk.35.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 321/ 363]                 blk.35.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 322/ 363]            blk.35.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 323/ 363]               blk.35.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 324/ 363]               blk.35.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 325/ 363]                 blk.35.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 326/ 363]              blk.35.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 327/ 363]               blk.35.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 328/ 363]                 blk.36.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 329/ 363]                 blk.36.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 330/ 363]                 blk.36.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 331/ 363]            blk.36.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 332/ 363]               blk.36.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 333/ 363]               blk.36.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 334/ 363]                 blk.36.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 335/ 363]              blk.36.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 336/ 363]               blk.36.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 337/ 363]                 blk.37.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 338/ 363]                 blk.37.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 339/ 363]                 blk.37.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 340/ 363]            blk.37.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 341/ 363]               blk.37.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 342/ 363]               blk.37.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 343/ 363]                 blk.37.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 344/ 363]              blk.37.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 345/ 363]               blk.37.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 346/ 363]                 blk.38.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 347/ 363]                 blk.38.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 348/ 363]                 blk.38.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 349/ 363]            blk.38.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 350/ 363]               blk.38.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 351/ 363]               blk.38.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 352/ 363]                 blk.38.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 353/ 363]              blk.38.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 354/ 363]               blk.38.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 355/ 363]                 blk.39.attn_q.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 356/ 363]                 blk.39.attn_k.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 357/ 363]                 blk.39.attn_v.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.120 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 358/ 363]            blk.39.attn_output.weight - [ 5120,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =    50.00 MiB ->    14.06 MiB | hist: 0.036 0.015 0.025 0.038 0.055 0.076 0.096 0.113 0.121 0.113 0.096 0.076 0.055 0.038 0.025 0.021 \n",
      "[ 359/ 363]               blk.39.ffn_gate.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 360/ 363]               blk.39.ffn_down.weight - [13824,  5120,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.015 0.024 0.037 0.054 0.076 0.098 0.115 0.122 0.115 0.098 0.076 0.054 0.037 0.024 0.020 \n",
      "[ 361/ 363]                 blk.39.ffn_up.weight - [ 5120, 13824,     1,     1], type =    f16, quantizing to q4_0 .. size =   135.00 MiB ->    37.97 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 362/ 363]              blk.39.attn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "[ 363/ 363]               blk.39.ffn_norm.weight - [ 5120,     1,     1,     1], type =    f32, size =    0.020 MB\n",
      "llama_model_quantize_internal: model size  = 24826.58 MB\n",
      "llama_model_quantize_internal: quant size  =  7023.90 MB\n",
      "llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "\n",
      "main: quantize time = 33890.21 ms\n",
      "main:    total time = 33890.21 ms\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: quantizing './models/70B-v2/ggml-model-f16.gguf' to './models/70B-v2/ggml-model-q4_0.gguf' as Q4_0\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type  f16:  562 tensors\n",
      "llama_model_quantize_internal: meta size = 1740160 bytes\n",
      "[   1/ 723]                    token_embd.weight - [ 8192, 32000,     1,     1], type =    f16, quantizing to q4_0 .. size =   500.00 MiB ->   140.62 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[   2/ 723]                   output_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[   3/ 723]                        output.weight - [ 8192, 32000,     1,     1], type =    f16, quantizing to q6_K .. size =   500.00 MiB ->   205.08 MiB | hist: \n",
      "[   4/ 723]                  blk.0.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.034 0.009 0.014 0.023 0.037 0.059 0.093 0.147 0.198 0.148 0.093 0.059 0.037 0.023 0.014 0.012 \n",
      "[   5/ 723]                  blk.0.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.034 0.008 0.013 0.021 0.035 0.057 0.094 0.153 0.201 0.153 0.094 0.057 0.035 0.021 0.013 0.011 \n",
      "[   6/ 723]                  blk.0.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.096 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[   7/ 723]             blk.0.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.022 0.034 0.052 0.074 0.099 0.120 0.128 0.120 0.099 0.075 0.052 0.035 0.022 0.018 \n",
      "[   8/ 723]                blk.0.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[   9/ 723]                blk.0.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.075 0.098 0.117 0.125 0.117 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[  10/ 723]                  blk.0.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  11/ 723]               blk.0.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  12/ 723]                blk.0.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  13/ 723]                  blk.1.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.035 0.011 0.017 0.028 0.043 0.066 0.099 0.137 0.160 0.137 0.099 0.066 0.043 0.028 0.017 0.015 \n",
      "[  14/ 723]                  blk.1.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.013 0.021 0.033 0.050 0.073 0.099 0.124 0.135 0.124 0.099 0.073 0.050 0.033 0.021 0.018 \n",
      "[  15/ 723]                  blk.1.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.022 0.033 0.050 0.071 0.097 0.124 0.137 0.124 0.097 0.071 0.050 0.033 0.022 0.018 \n",
      "[  16/ 723]             blk.1.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.117 0.098 0.076 0.054 0.036 0.023 0.019 \n",
      "[  17/ 723]                blk.1.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  18/ 723]                blk.1.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  19/ 723]                  blk.1.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  20/ 723]               blk.1.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  21/ 723]                blk.1.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  22/ 723]                  blk.2.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.022 0.035 0.052 0.075 0.099 0.119 0.127 0.119 0.099 0.075 0.053 0.035 0.022 0.018 \n",
      "[  23/ 723]                  blk.2.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.098 0.117 0.125 0.117 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[  24/ 723]                  blk.2.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.116 0.098 0.075 0.054 0.036 0.023 0.019 \n",
      "[  25/ 723]             blk.2.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  26/ 723]                blk.2.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  27/ 723]                blk.2.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  28/ 723]                  blk.2.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  29/ 723]               blk.2.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  30/ 723]                blk.2.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  31/ 723]                  blk.3.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  32/ 723]                  blk.3.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.120 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  33/ 723]                  blk.3.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  34/ 723]             blk.3.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  35/ 723]                blk.3.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  36/ 723]                blk.3.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  37/ 723]                  blk.3.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  38/ 723]               blk.3.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  39/ 723]                blk.3.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  40/ 723]                  blk.4.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[  41/ 723]                  blk.4.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.024 0.020 \n",
      "[  42/ 723]                  blk.4.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.023 0.036 0.054 0.075 0.098 0.116 0.124 0.116 0.098 0.075 0.054 0.036 0.024 0.020 \n",
      "[  43/ 723]             blk.4.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  44/ 723]                blk.4.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  45/ 723]                blk.4.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  46/ 723]                  blk.4.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  47/ 723]               blk.4.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  48/ 723]                blk.4.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  49/ 723]                  blk.5.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  50/ 723]                  blk.5.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  51/ 723]                  blk.5.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[  52/ 723]             blk.5.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  53/ 723]                blk.5.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  54/ 723]                blk.5.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  55/ 723]                  blk.5.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  56/ 723]               blk.5.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  57/ 723]                blk.5.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  58/ 723]                  blk.6.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  59/ 723]                  blk.6.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  60/ 723]                  blk.6.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.055 0.038 0.025 0.020 \n",
      "[  61/ 723]             blk.6.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  62/ 723]                blk.6.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  63/ 723]                blk.6.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  64/ 723]                  blk.6.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  65/ 723]               blk.6.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  66/ 723]                blk.6.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  67/ 723]                  blk.7.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  68/ 723]                  blk.7.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  69/ 723]                  blk.7.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  70/ 723]             blk.7.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  71/ 723]                blk.7.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  72/ 723]                blk.7.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  73/ 723]                  blk.7.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  74/ 723]               blk.7.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  75/ 723]                blk.7.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  76/ 723]                  blk.8.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  77/ 723]                  blk.8.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  78/ 723]                  blk.8.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[  79/ 723]             blk.8.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  80/ 723]                blk.8.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  81/ 723]                blk.8.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  82/ 723]                  blk.8.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  83/ 723]               blk.8.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  84/ 723]                blk.8.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  85/ 723]                  blk.9.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[  86/ 723]                  blk.9.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  87/ 723]                  blk.9.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[  88/ 723]             blk.9.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  89/ 723]                blk.9.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  90/ 723]                blk.9.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[  91/ 723]                  blk.9.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  92/ 723]               blk.9.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  93/ 723]                blk.9.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[  94/ 723]                 blk.10.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[  95/ 723]                 blk.10.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[  96/ 723]                 blk.10.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[  97/ 723]            blk.10.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[  98/ 723]               blk.10.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[  99/ 723]               blk.10.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 100/ 723]                 blk.10.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 101/ 723]              blk.10.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 102/ 723]               blk.10.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 103/ 723]                 blk.11.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 104/ 723]                 blk.11.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 105/ 723]                 blk.11.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 106/ 723]            blk.11.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 107/ 723]               blk.11.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 108/ 723]               blk.11.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 109/ 723]                 blk.11.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 110/ 723]              blk.11.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 111/ 723]               blk.11.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 112/ 723]                 blk.12.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 113/ 723]                 blk.12.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 114/ 723]                 blk.12.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 115/ 723]            blk.12.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 116/ 723]               blk.12.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 117/ 723]               blk.12.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 118/ 723]                 blk.12.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 119/ 723]              blk.12.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 120/ 723]               blk.12.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 121/ 723]                 blk.13.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 122/ 723]                 blk.13.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 123/ 723]                 blk.13.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 124/ 723]            blk.13.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 125/ 723]               blk.13.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 126/ 723]               blk.13.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 127/ 723]                 blk.13.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 128/ 723]              blk.13.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 129/ 723]               blk.13.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 130/ 723]                 blk.14.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 131/ 723]                 blk.14.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 132/ 723]                 blk.14.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 133/ 723]            blk.14.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 134/ 723]               blk.14.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 135/ 723]               blk.14.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 136/ 723]                 blk.14.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 137/ 723]              blk.14.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 138/ 723]               blk.14.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 139/ 723]                 blk.15.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 140/ 723]                 blk.15.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 141/ 723]                 blk.15.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 142/ 723]            blk.15.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 143/ 723]               blk.15.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 144/ 723]               blk.15.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 145/ 723]                 blk.15.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 146/ 723]              blk.15.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 147/ 723]               blk.15.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 148/ 723]                 blk.16.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 149/ 723]                 blk.16.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 150/ 723]                 blk.16.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 151/ 723]            blk.16.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 152/ 723]               blk.16.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 153/ 723]               blk.16.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 154/ 723]                 blk.16.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 155/ 723]              blk.16.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 156/ 723]               blk.16.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 157/ 723]                 blk.17.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 158/ 723]                 blk.17.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 159/ 723]                 blk.17.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 160/ 723]            blk.17.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 161/ 723]               blk.17.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 162/ 723]               blk.17.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 163/ 723]                 blk.17.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 164/ 723]              blk.17.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 165/ 723]               blk.17.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 166/ 723]                 blk.18.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 167/ 723]                 blk.18.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 168/ 723]                 blk.18.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 169/ 723]            blk.18.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 170/ 723]               blk.18.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 171/ 723]               blk.18.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 172/ 723]                 blk.18.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 173/ 723]              blk.18.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 174/ 723]               blk.18.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 175/ 723]                 blk.19.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 176/ 723]                 blk.19.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 177/ 723]                 blk.19.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 178/ 723]            blk.19.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 179/ 723]               blk.19.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 180/ 723]               blk.19.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 181/ 723]                 blk.19.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 182/ 723]              blk.19.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 183/ 723]               blk.19.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 184/ 723]                 blk.20.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 185/ 723]                 blk.20.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 186/ 723]                 blk.20.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.111 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 187/ 723]            blk.20.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 188/ 723]               blk.20.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 189/ 723]               blk.20.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 190/ 723]                 blk.20.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 191/ 723]              blk.20.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 192/ 723]               blk.20.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 193/ 723]                 blk.21.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 194/ 723]                 blk.21.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 195/ 723]                 blk.21.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 196/ 723]            blk.21.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 197/ 723]               blk.21.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 198/ 723]               blk.21.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 199/ 723]                 blk.21.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 200/ 723]              blk.21.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 201/ 723]               blk.21.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 202/ 723]                 blk.22.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 203/ 723]                 blk.22.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 204/ 723]                 blk.22.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 205/ 723]            blk.22.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 206/ 723]               blk.22.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 207/ 723]               blk.22.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 208/ 723]                 blk.22.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 209/ 723]              blk.22.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 210/ 723]               blk.22.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 211/ 723]                 blk.23.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 212/ 723]                 blk.23.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 213/ 723]                 blk.23.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 214/ 723]            blk.23.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 215/ 723]               blk.23.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 216/ 723]               blk.23.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 217/ 723]                 blk.23.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 218/ 723]              blk.23.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 219/ 723]               blk.23.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 220/ 723]                 blk.24.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 221/ 723]                 blk.24.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 222/ 723]                 blk.24.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 223/ 723]            blk.24.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 224/ 723]               blk.24.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 225/ 723]               blk.24.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 226/ 723]                 blk.24.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 227/ 723]              blk.24.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 228/ 723]               blk.24.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 229/ 723]                 blk.25.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 230/ 723]                 blk.25.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 231/ 723]                 blk.25.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 232/ 723]            blk.25.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 233/ 723]               blk.25.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 234/ 723]               blk.25.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 235/ 723]                 blk.25.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 236/ 723]              blk.25.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 237/ 723]               blk.25.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 238/ 723]                 blk.26.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 239/ 723]                 blk.26.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 240/ 723]                 blk.26.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 241/ 723]            blk.26.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 242/ 723]               blk.26.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 243/ 723]               blk.26.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 244/ 723]                 blk.26.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 245/ 723]              blk.26.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 246/ 723]               blk.26.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 247/ 723]                 blk.27.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 248/ 723]                 blk.27.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 249/ 723]                 blk.27.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 250/ 723]            blk.27.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 251/ 723]               blk.27.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 252/ 723]               blk.27.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 253/ 723]                 blk.27.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 254/ 723]              blk.27.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 255/ 723]               blk.27.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 256/ 723]                 blk.28.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 257/ 723]                 blk.28.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 258/ 723]                 blk.28.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 259/ 723]            blk.28.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 260/ 723]               blk.28.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 261/ 723]               blk.28.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 262/ 723]                 blk.28.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 263/ 723]              blk.28.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 264/ 723]               blk.28.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 265/ 723]                 blk.29.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 266/ 723]                 blk.29.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 267/ 723]                 blk.29.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 268/ 723]            blk.29.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 269/ 723]               blk.29.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 270/ 723]               blk.29.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 271/ 723]                 blk.29.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 272/ 723]              blk.29.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 273/ 723]               blk.29.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 274/ 723]                 blk.30.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 275/ 723]                 blk.30.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 276/ 723]                 blk.30.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 277/ 723]            blk.30.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 278/ 723]               blk.30.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 279/ 723]               blk.30.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 280/ 723]                 blk.30.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 281/ 723]              blk.30.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 282/ 723]               blk.30.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 283/ 723]                 blk.31.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 284/ 723]                 blk.31.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 285/ 723]                 blk.31.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 286/ 723]            blk.31.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 287/ 723]               blk.31.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 288/ 723]               blk.31.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 289/ 723]                 blk.31.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 290/ 723]              blk.31.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 291/ 723]               blk.31.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 292/ 723]                 blk.32.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 293/ 723]                 blk.32.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 294/ 723]                 blk.32.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 295/ 723]            blk.32.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 296/ 723]               blk.32.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 297/ 723]               blk.32.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 298/ 723]                 blk.32.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 299/ 723]              blk.32.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 300/ 723]               blk.32.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 301/ 723]                 blk.33.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 302/ 723]                 blk.33.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 303/ 723]                 blk.33.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 304/ 723]            blk.33.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 305/ 723]               blk.33.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 306/ 723]               blk.33.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 307/ 723]                 blk.33.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 308/ 723]              blk.33.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 309/ 723]               blk.33.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 310/ 723]                 blk.34.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 311/ 723]                 blk.34.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 312/ 723]                 blk.34.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 313/ 723]            blk.34.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 314/ 723]               blk.34.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 315/ 723]               blk.34.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 316/ 723]                 blk.34.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 317/ 723]              blk.34.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 318/ 723]               blk.34.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 319/ 723]                 blk.35.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 320/ 723]                 blk.35.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 321/ 723]                 blk.35.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 322/ 723]            blk.35.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 323/ 723]               blk.35.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 324/ 723]               blk.35.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 325/ 723]                 blk.35.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 326/ 723]              blk.35.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 327/ 723]               blk.35.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 328/ 723]                 blk.36.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 329/ 723]                 blk.36.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 330/ 723]                 blk.36.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 331/ 723]            blk.36.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 332/ 723]               blk.36.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 333/ 723]               blk.36.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 334/ 723]                 blk.36.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 335/ 723]              blk.36.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 336/ 723]               blk.36.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 337/ 723]                 blk.37.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 338/ 723]                 blk.37.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 339/ 723]                 blk.37.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.076 0.057 0.039 0.025 0.021 \n",
      "[ 340/ 723]            blk.37.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 341/ 723]               blk.37.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 342/ 723]               blk.37.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 343/ 723]                 blk.37.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 344/ 723]              blk.37.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 345/ 723]               blk.37.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 346/ 723]                 blk.38.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 347/ 723]                 blk.38.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 348/ 723]                 blk.38.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 349/ 723]            blk.38.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 350/ 723]               blk.38.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 351/ 723]               blk.38.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 352/ 723]                 blk.38.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 353/ 723]              blk.38.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 354/ 723]               blk.38.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 355/ 723]                 blk.39.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 356/ 723]                 blk.39.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 357/ 723]                 blk.39.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 358/ 723]            blk.39.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 359/ 723]               blk.39.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 360/ 723]               blk.39.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 361/ 723]                 blk.39.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 362/ 723]              blk.39.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 363/ 723]               blk.39.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 364/ 723]                 blk.40.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 365/ 723]                 blk.40.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 366/ 723]                 blk.40.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 367/ 723]            blk.40.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 368/ 723]               blk.40.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 369/ 723]               blk.40.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 370/ 723]                 blk.40.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 371/ 723]              blk.40.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 372/ 723]               blk.40.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 373/ 723]                 blk.41.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 374/ 723]                 blk.41.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 375/ 723]                 blk.41.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 376/ 723]            blk.41.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 377/ 723]               blk.41.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 378/ 723]               blk.41.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 379/ 723]                 blk.41.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 380/ 723]              blk.41.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 381/ 723]               blk.41.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 382/ 723]                 blk.42.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 383/ 723]                 blk.42.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 384/ 723]                 blk.42.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 385/ 723]            blk.42.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 386/ 723]               blk.42.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 387/ 723]               blk.42.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 388/ 723]                 blk.42.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 389/ 723]              blk.42.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 390/ 723]               blk.42.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 391/ 723]                 blk.43.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 392/ 723]                 blk.43.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 393/ 723]                 blk.43.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 394/ 723]            blk.43.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 395/ 723]               blk.43.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 396/ 723]               blk.43.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 397/ 723]                 blk.43.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 398/ 723]              blk.43.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 399/ 723]               blk.43.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 400/ 723]                 blk.44.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 401/ 723]                 blk.44.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 402/ 723]                 blk.44.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 403/ 723]            blk.44.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 404/ 723]               blk.44.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 405/ 723]               blk.44.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 406/ 723]                 blk.44.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 407/ 723]              blk.44.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 408/ 723]               blk.44.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 409/ 723]                 blk.45.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 410/ 723]                 blk.45.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 411/ 723]                 blk.45.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 412/ 723]            blk.45.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 413/ 723]               blk.45.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 414/ 723]               blk.45.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 415/ 723]                 blk.45.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 416/ 723]              blk.45.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 417/ 723]               blk.45.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 418/ 723]                 blk.46.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 419/ 723]                 blk.46.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 420/ 723]                 blk.46.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.111 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 421/ 723]            blk.46.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 422/ 723]               blk.46.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 423/ 723]               blk.46.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 424/ 723]                 blk.46.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 425/ 723]              blk.46.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 426/ 723]               blk.46.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 427/ 723]                 blk.47.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 428/ 723]                 blk.47.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.039 0.025 0.020 \n",
      "[ 429/ 723]                 blk.47.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 430/ 723]            blk.47.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 431/ 723]               blk.47.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 432/ 723]               blk.47.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 433/ 723]                 blk.47.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 434/ 723]              blk.47.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 435/ 723]               blk.47.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 436/ 723]                 blk.48.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 437/ 723]                 blk.48.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.025 0.020 \n",
      "[ 438/ 723]                 blk.48.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 439/ 723]            blk.48.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 440/ 723]               blk.48.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 441/ 723]               blk.48.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 442/ 723]                 blk.48.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 443/ 723]              blk.48.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 444/ 723]               blk.48.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 445/ 723]                 blk.49.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 446/ 723]                 blk.49.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 447/ 723]                 blk.49.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 448/ 723]            blk.49.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 449/ 723]               blk.49.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 450/ 723]               blk.49.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 451/ 723]                 blk.49.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 452/ 723]              blk.49.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 453/ 723]               blk.49.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 454/ 723]                 blk.50.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 455/ 723]                 blk.50.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.123 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 456/ 723]                 blk.50.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 457/ 723]            blk.50.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 458/ 723]               blk.50.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 459/ 723]               blk.50.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 460/ 723]                 blk.50.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 461/ 723]              blk.50.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 462/ 723]               blk.50.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 463/ 723]                 blk.51.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 464/ 723]                 blk.51.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 465/ 723]                 blk.51.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 466/ 723]            blk.51.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 467/ 723]               blk.51.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 468/ 723]               blk.51.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 469/ 723]                 blk.51.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 470/ 723]              blk.51.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 471/ 723]               blk.51.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 472/ 723]                 blk.52.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 473/ 723]                 blk.52.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.077 0.056 0.038 0.025 0.020 \n",
      "[ 474/ 723]                 blk.52.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 475/ 723]            blk.52.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 476/ 723]               blk.52.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 477/ 723]               blk.52.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 478/ 723]                 blk.52.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 479/ 723]              blk.52.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 480/ 723]               blk.52.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 481/ 723]                 blk.53.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 482/ 723]                 blk.53.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[ 483/ 723]                 blk.53.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 484/ 723]            blk.53.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 485/ 723]               blk.53.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 486/ 723]               blk.53.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 487/ 723]                 blk.53.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 488/ 723]              blk.53.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 489/ 723]               blk.53.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 490/ 723]                 blk.54.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 491/ 723]                 blk.54.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[ 492/ 723]                 blk.54.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 493/ 723]            blk.54.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 494/ 723]               blk.54.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 495/ 723]               blk.54.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 496/ 723]                 blk.54.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 497/ 723]              blk.54.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 498/ 723]               blk.54.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 499/ 723]                 blk.55.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 500/ 723]                 blk.55.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 501/ 723]                 blk.55.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 502/ 723]            blk.55.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 503/ 723]               blk.55.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 504/ 723]               blk.55.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 505/ 723]                 blk.55.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 506/ 723]              blk.55.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 507/ 723]               blk.55.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 508/ 723]                 blk.56.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 509/ 723]                 blk.56.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 510/ 723]                 blk.56.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 511/ 723]            blk.56.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 512/ 723]               blk.56.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 513/ 723]               blk.56.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 514/ 723]                 blk.56.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 515/ 723]              blk.56.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 516/ 723]               blk.56.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 517/ 723]                 blk.57.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 518/ 723]                 blk.57.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.075 0.055 0.037 0.024 0.020 \n",
      "[ 519/ 723]                 blk.57.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.076 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 520/ 723]            blk.57.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 521/ 723]               blk.57.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 522/ 723]               blk.57.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 523/ 723]                 blk.57.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 524/ 723]              blk.57.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 525/ 723]               blk.57.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 526/ 723]                 blk.58.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 527/ 723]                 blk.58.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.023 0.036 0.054 0.075 0.097 0.117 0.126 0.117 0.097 0.075 0.054 0.037 0.023 0.019 \n",
      "[ 528/ 723]                 blk.58.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 529/ 723]            blk.58.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 530/ 723]               blk.58.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 531/ 723]               blk.58.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 532/ 723]                 blk.58.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 533/ 723]              blk.58.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 534/ 723]               blk.58.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 535/ 723]                 blk.59.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 536/ 723]                 blk.59.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.037 0.054 0.075 0.097 0.116 0.125 0.116 0.097 0.075 0.054 0.037 0.024 0.019 \n",
      "[ 537/ 723]                 blk.59.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 538/ 723]            blk.59.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 539/ 723]               blk.59.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 540/ 723]               blk.59.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 541/ 723]                 blk.59.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 542/ 723]              blk.59.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 543/ 723]               blk.59.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 544/ 723]                 blk.60.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.037 0.054 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 545/ 723]                 blk.60.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.013 0.021 0.033 0.050 0.072 0.097 0.123 0.140 0.123 0.097 0.072 0.050 0.034 0.021 0.018 \n",
      "[ 546/ 723]                 blk.60.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 547/ 723]            blk.60.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 548/ 723]               blk.60.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 549/ 723]               blk.60.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 550/ 723]                 blk.60.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 551/ 723]              blk.60.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 552/ 723]               blk.60.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 553/ 723]                 blk.61.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 554/ 723]                 blk.61.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.074 0.097 0.118 0.129 0.118 0.097 0.074 0.053 0.036 0.023 0.019 \n",
      "[ 555/ 723]                 blk.61.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 556/ 723]            blk.61.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 557/ 723]               blk.61.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 558/ 723]               blk.61.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 559/ 723]                 blk.61.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 560/ 723]              blk.61.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 561/ 723]               blk.61.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 562/ 723]                 blk.62.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 563/ 723]                 blk.62.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.074 0.097 0.118 0.129 0.118 0.097 0.074 0.053 0.036 0.023 0.019 \n",
      "[ 564/ 723]                 blk.62.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 565/ 723]            blk.62.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 566/ 723]               blk.62.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 567/ 723]               blk.62.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 568/ 723]                 blk.62.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 569/ 723]              blk.62.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 570/ 723]               blk.62.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 571/ 723]                 blk.63.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 572/ 723]                 blk.63.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.035 0.052 0.074 0.097 0.119 0.132 0.119 0.097 0.074 0.053 0.035 0.023 0.019 \n",
      "[ 573/ 723]                 blk.63.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 574/ 723]            blk.63.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 575/ 723]               blk.63.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 576/ 723]               blk.63.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 577/ 723]                 blk.63.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 578/ 723]              blk.63.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 579/ 723]               blk.63.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 580/ 723]                 blk.64.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 581/ 723]                 blk.64.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.036 0.054 0.075 0.097 0.117 0.125 0.117 0.097 0.075 0.054 0.037 0.024 0.019 \n",
      "[ 582/ 723]                 blk.64.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 583/ 723]            blk.64.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 584/ 723]               blk.64.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 585/ 723]               blk.64.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 586/ 723]                 blk.64.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 587/ 723]              blk.64.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 588/ 723]               blk.64.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 589/ 723]                 blk.65.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.115 0.122 0.115 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 590/ 723]                 blk.65.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.013 0.022 0.034 0.051 0.072 0.097 0.122 0.138 0.122 0.097 0.072 0.051 0.034 0.022 0.018 \n",
      "[ 591/ 723]                 blk.65.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 592/ 723]            blk.65.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 593/ 723]               blk.65.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 594/ 723]               blk.65.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 595/ 723]                 blk.65.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 596/ 723]              blk.65.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 597/ 723]               blk.65.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 598/ 723]                 blk.66.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.120 0.113 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 599/ 723]                 blk.66.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.036 0.053 0.074 0.097 0.119 0.130 0.119 0.097 0.074 0.053 0.035 0.023 0.019 \n",
      "[ 600/ 723]                 blk.66.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.112 0.117 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 601/ 723]            blk.66.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 602/ 723]               blk.66.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 603/ 723]               blk.66.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 604/ 723]                 blk.66.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 605/ 723]              blk.66.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 606/ 723]               blk.66.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 607/ 723]                 blk.67.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.014 0.023 0.036 0.054 0.075 0.097 0.116 0.125 0.117 0.097 0.075 0.054 0.036 0.023 0.019 \n",
      "[ 608/ 723]                 blk.67.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.014 0.023 0.035 0.053 0.074 0.097 0.119 0.133 0.119 0.096 0.073 0.052 0.035 0.023 0.019 \n",
      "[ 609/ 723]                 blk.67.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 610/ 723]            blk.67.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 611/ 723]               blk.67.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 612/ 723]               blk.67.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 613/ 723]                 blk.67.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 614/ 723]              blk.67.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 615/ 723]               blk.67.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 616/ 723]                 blk.68.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 617/ 723]                 blk.68.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 618/ 723]                 blk.68.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 619/ 723]            blk.68.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 620/ 723]               blk.68.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 621/ 723]               blk.68.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 622/ 723]                 blk.68.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 623/ 723]              blk.68.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 624/ 723]               blk.68.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 625/ 723]                 blk.69.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 626/ 723]                 blk.69.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.113 0.120 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 627/ 723]                 blk.69.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 628/ 723]            blk.69.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 629/ 723]               blk.69.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 630/ 723]               blk.69.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 631/ 723]                 blk.69.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 632/ 723]              blk.69.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 633/ 723]               blk.69.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 634/ 723]                 blk.70.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 635/ 723]                 blk.70.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.024 0.038 0.055 0.076 0.097 0.114 0.121 0.114 0.097 0.076 0.055 0.038 0.024 0.020 \n",
      "[ 636/ 723]                 blk.70.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.037 0.016 0.025 0.039 0.056 0.076 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 637/ 723]            blk.70.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 638/ 723]               blk.70.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 639/ 723]               blk.70.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 640/ 723]                 blk.70.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 641/ 723]              blk.70.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 642/ 723]               blk.70.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 643/ 723]                 blk.71.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 644/ 723]                 blk.71.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 645/ 723]                 blk.71.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.111 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 646/ 723]            blk.71.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 647/ 723]               blk.71.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 648/ 723]               blk.71.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 649/ 723]                 blk.71.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 650/ 723]              blk.71.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 651/ 723]               blk.71.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 652/ 723]                 blk.72.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 653/ 723]                 blk.72.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 654/ 723]                 blk.72.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 655/ 723]            blk.72.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 656/ 723]               blk.72.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 657/ 723]               blk.72.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 658/ 723]                 blk.72.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 659/ 723]              blk.72.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 660/ 723]               blk.72.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 661/ 723]                 blk.73.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 662/ 723]                 blk.73.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 663/ 723]                 blk.73.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 664/ 723]            blk.73.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 665/ 723]               blk.73.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 666/ 723]               blk.73.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 667/ 723]                 blk.73.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 668/ 723]              blk.73.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 669/ 723]               blk.73.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 670/ 723]                 blk.74.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 671/ 723]                 blk.74.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 672/ 723]                 blk.74.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 673/ 723]            blk.74.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.115 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 674/ 723]               blk.74.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 675/ 723]               blk.74.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 676/ 723]                 blk.74.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 677/ 723]              blk.74.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 678/ 723]               blk.74.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 679/ 723]                 blk.75.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 680/ 723]                 blk.75.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 681/ 723]                 blk.75.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 682/ 723]            blk.75.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 683/ 723]               blk.75.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 684/ 723]               blk.75.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 685/ 723]                 blk.75.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 686/ 723]              blk.75.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 687/ 723]               blk.75.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 688/ 723]                 blk.76.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 689/ 723]                 blk.76.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.076 0.096 0.112 0.118 0.112 0.097 0.077 0.056 0.038 0.025 0.021 \n",
      "[ 690/ 723]                 blk.76.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 691/ 723]            blk.76.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 692/ 723]               blk.76.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 693/ 723]               blk.76.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 694/ 723]                 blk.76.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 695/ 723]              blk.76.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 696/ 723]               blk.76.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 697/ 723]                 blk.77.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 698/ 723]                 blk.77.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 699/ 723]                 blk.77.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 700/ 723]            blk.77.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.040 0.057 0.077 0.096 0.110 0.115 0.110 0.096 0.077 0.057 0.040 0.026 0.021 \n",
      "[ 701/ 723]               blk.77.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 702/ 723]               blk.77.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 703/ 723]                 blk.77.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 704/ 723]              blk.77.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 705/ 723]               blk.77.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 706/ 723]                 blk.78.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.039 0.025 0.021 \n",
      "[ 707/ 723]                 blk.78.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.113 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 708/ 723]                 blk.78.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 709/ 723]            blk.78.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.026 0.021 \n",
      "[ 710/ 723]               blk.78.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 711/ 723]               blk.78.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 712/ 723]                 blk.78.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.097 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "[ 713/ 723]              blk.78.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 714/ 723]               blk.78.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 715/ 723]                 blk.79.attn_q.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.118 0.112 0.097 0.076 0.056 0.038 0.025 0.021 \n",
      "[ 716/ 723]                 blk.79.attn_k.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 717/ 723]                 blk.79.attn_v.weight - [ 8192,  1024,     1,     1], type =    f16, quantizing to q4_0 .. size =    16.00 MiB ->     4.50 MiB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.112 0.119 0.112 0.097 0.076 0.056 0.038 0.025 0.020 \n",
      "[ 718/ 723]            blk.79.attn_output.weight - [ 8192,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   128.00 MiB ->    36.00 MiB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.115 0.110 0.096 0.077 0.057 0.040 0.026 0.021 \n",
      "[ 719/ 723]               blk.79.ffn_gate.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.097 0.111 0.117 0.111 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 720/ 723]               blk.79.ffn_down.weight - [28672,  8192,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.024 0.037 0.055 0.076 0.097 0.114 0.122 0.114 0.097 0.076 0.055 0.037 0.024 0.020 \n",
      "[ 721/ 723]                 blk.79.ffn_up.weight - [ 8192, 28672,     1,     1], type =    f16, quantizing to q4_0 .. size =   448.00 MiB ->   126.00 MiB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.097 0.112 0.117 0.112 0.097 0.077 0.056 0.039 0.025 0.021 \n",
      "[ 722/ 723]              blk.79.attn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "[ 723/ 723]               blk.79.ffn_norm.weight - [ 8192,     1,     1,     1], type =    f32, size =    0.031 MB\n",
      "llama_model_quantize_internal: model size  = 131565.03 MB\n",
      "llama_model_quantize_internal: quant size  = 37070.73 MB\n",
      "llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021 \n",
      "\n",
      "main: quantize time = 202600.20 ms\n",
      "main:    total time = 202600.20 ms\n"
     ]
    }
   ],
   "source": [
    "# quantize the model to 4-bits (using q4_0 method)\n",
    "!./quantize ./models/7B-v2/ggml-model-f16.gguf ./models/7B-v2/ggml-model-q4_0.gguf q4_0\n",
    "!./quantize ./models/13B-v2/ggml-model-f16.gguf ./models/13B-v2/ggml-model-q4_0.gguf q4_0\n",
    "!./quantize ./models/70B-v2/ggml-model-f16.gguf ./models/70B-v2/ggml-model-q4_0.gguf q4_0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "416ca561-de1a-4094-ae0b-fd71408d45e6",
   "metadata": {},
   "source": [
    "# inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f0aede9-2f19-41e1-bb4f-1a1d30a00156",
   "metadata": {},
   "source": [
    "### 7B Q4_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4c50d2ab-fc82-4119-8ac3-38ead2b8fee8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314358\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_0:  225 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   70.42 MiB\n",
      "llm_load_tensors: VRAM used           = 3577.55 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "..................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 3904.05 MiB (model: 3577.55 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her star, she conquered and peopled half the world at once; for like a rough matron full of benevolence, she had emigrated all her children anyways. \n",
      " England is not only the centre of the English-speaking Union, but the home of all that is best worth reading in English, French or any other language.\n",
      "England was not only the home of Shakespeare and Milton and the Bible, it also housed many famous authors from around the world including Mark Twain, Alexandre Dumas, Jack London, Walt Whitman, Edgar Rice Burroughs and more... \n",
      "A collection of 423 literary masterpieces from around the globe in English and many other languages! In this amazing 1065 page anthology, you will find an enormous treasure trove of reading pleasure for everyone who likes to read. With the addition of more than a hundred books that have been translated into English from other languages, there are now over four-hundred famous classics and works by authors from around the world to choose from!\n",
      "Anthology Table of Contents:\n",
      "1 - A Thousand Miles in a Balloon - Jules Verne\n",
      "2 - Alice's Adventures In Wonderland - Lewis Carroll\n",
      "3 - Anne Of Green Gables - L.M. Montgomery\n",
      "4 - Arabian Nights And Days - Various Authors\n",
      "5 - Around The World In Eighty Days - Jules Verne\n",
      "6 - A Christmas Carol - Charles Dickens\n",
      "7 - A Connecticut Yankee in King Arthur's Court - Mark Twain\n",
      "8 - Aladdin & The Enchanted Lamp - W.A. Cooksey (Translation)\n",
      "9 - Alice Through The Looking Glass - Lewis Carroll\n",
      "10 - Ancient Mariner & Other Poems - Coleridge, S.T.\n",
      "11 - Anna Karenina - Leo Tolstoy\n",
      "12 - Arabian Nights And Days - Various Authors\n",
      "13 - Arabian Nights Entertainments - Various Authors\n",
      "14 - Arabian Tales & Fables - Anonymous (Translation)\n",
      "15 - Around The World In 80 Days - Jules Verne\n",
      "16 - Arthur Conan Doyle - A Collection Of Stories And Poems\n",
      "17 - Aesop's Fables - Various Authors\n",
      "18 - Attar - The Conference of Birds (Translation)\n",
      "19 - Bab Ballads & Other Verses - Eliza Cook\n",
      "20 - Barsoom Series By Edgar Rice Burroughs\n",
      "21 - Batman Stories Written by Bill Finger\n",
      "22 - Beelzebub & The Golden Calf - Gogol (Translation)\n",
      "23 - Ben Hur (The Classic Original Story By Lew Wallace)\n",
      "24 - Beside The Fire - A Collection Of Scottish Tales And Lore\n",
      "25 - Black Beauty - Anna Sewell\n",
      "26 - Blue Fairy Book - Andrew Lang\n",
      "27 - Bounder - Anonymous\n",
      "28 - Brahma Samhita - Various Authors (Translation)\n",
      "29 - Bram Stoker's Dracula & Other Tales Of Mystery And Horror\n",
      "30 - Breaking Out By Frank L. Baum\n",
      "31 - Broken Sword - Sir Walter Scott\n",
      "32 - Buchan's Journal - The Adventures Of Captain Hatteras (Translation)\n",
      "33 - Burning Wick - Various Authors\n",
      "34 - Canoe Man's Log - George A. Birmingham\n",
      "35 - Caperuccio By Ludovico Ariosto\n",
      "36 - Carmen, the Tragedy of a Ruse - Prosper Merimee (Translation)\n",
      "37 - Casanova - Memoir Of His Own Life, 14 Volumes\n",
      "38 - Chant Royal - Charles d'Orleans\n",
      "39 - Chevalier De Maison Rouge By Alain-Rene Lesage\n",
      "40 - Childrens' Stories In Many Languages\n",
      "41 - Chronicles of Narnia (7 Vol. Collection) by C.S. Lewis\n",
      "42 - Codex Regius: The Bible Of Saint Jerome (Translation)\n",
      "43 - Companion To The Works Of Shakespeare By Edmund Malone\n",
      "44 - Coningsby - Benjamin Disraeli\n",
      "45 - Confessions of an English Opium-Eater By Thomas de Quincey\n",
      "46 - Consolation By Boethius (Translated by Alfred Church)\n",
      "47 - Count Of Monte Cristo - Alexandre Dumas, Pere'\n",
      "4\n",
      "llama_print_timings:        load time =    3924.93 ms\n",
      "llama_print_timings:      sample time =     347.89 ms /  1024 runs   (    0.34 ms per token,  2943.48 tokens per second)\n",
      "llama_print_timings: prompt eval time =     131.27 ms /   494 tokens (    0.27 ms per token,  3763.15 tokens per second)\n",
      "llama_print_timings:        eval time =    8666.15 ms /  1023 runs   (    8.47 ms per token,   118.05 tokens per second)\n",
      "llama_print_timings:       total time =    9428.35 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3e237938-4375-406a-b329-48d6d2363aa9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314372\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_0:  225 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   70.42 MiB\n",
      "llm_load_tensors: VRAM used           = 3577.55 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "..................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 3904.05 MiB (model: 3577.55 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her sprightly minister, the French nation are undergoing a change which gives promise of introducing a better love-making into all countries, as well as providing subject for a fine new book of sermons. I have but an indifferent taste in such matters, or I would quote the passage about Rouen: \"A fine city--the Seine above in full flood, the cathedral spire climbing like a column of smoke above half-a-dozen steeples.\" \n",
      " It is the history of France and England, that has lately been run in almost as many parts as there are newspapers. Those nations (especially France) which have endeavoured to imitate them, have suffered severely from the pressures of an unaffected simplicity upon their enlightened minds. \n",
      " The Red-coat troops being removed on Tuesday and Wednesday last from the metropolis: on Thursday se'night I took a walk out of town, to see what had been done in that part, and whether any change could be discerned or noticed. \n",
      " As I was walking westward along the edge of Hyde Park, my attention was attracted by a very large and handsome building, which lay close at hand: it was intended for a hotel of moderate pretensions. \n",
      " But as there are many buildings of that kind in London, I did not take special notice of it: however I turned in and went into the yard, where I saw a large brick wall running round three sides; on the fourth side was an entrance door. \n",
      " The building itself was not very conspicuous from without; but the entrance gate appeared to be a good specimen of that kind of work, for it was covered all over with a profusion of brass-work; so much so indeed as almost to give it a gilded appearance. \n",
      " But when I entered this large and handsome building, my attention was at once arrested by a great variety of banners, standards, streamers, flags and pennants flying from every window--and especially on the principal façade, in such manner as to cause it to look like one continued triumphal arch: notwithstanding which I did not perceive anything particular in the architecture of the building, excepting that its style was a little florid. \n",
      " At length (as I was passing along the hall) I discovered in an inner window--at no great height above my head, two small brass figures at the extremity of a flag-staff; each figure bearing a small flag, and both holding long swords in their right hands pointed to heaven. \n",
      " At the same time one of these figures being turned towards me, I saw (by the light of an extinguished candle) that it was no other personage than Father Thick--with his arms extended towards heaven; as if he were making a most earnest prayer for deliverance from some very imminent danger. \n",
      " This figure immediately struck my attention: but as I passed onwards and entered the next room (which had a window at one side of it, that afforded me another view of him) I saw that he was standing with his arms out-stretched in the same manner, though there were several candles lighted round him; so that he was clearly to be seen--and in such an attitude as to give him a most singular and ludicrous appearance. \n",
      " As this was the first view I had of Father Thick in his proper personage since my arrival at Paris--I took notice that he seemed to have been greatly altered for the worse, from what he used to be, by the loss of all his supernumerary members; and the more so as he appeared in a very different character and situation than when I last saw him. \n",
      " As for himself (being not only disfigured but deformed) there was no appearance of anything in the least like humanity left about him: except that it was impossible to avoid taking notice of his resemblance both to a wild beast, as well as a monster of another kind--which I thought rather more than any other particular resemblance. \n",
      "Besides which (being all these things) he had been the death and misery of several poor people: being likewise the means of ruining my own reputation among respectable people--as well as a great deal of trouble to me in different ways, though in some points not to be accounted for. \n",
      " In short I found him a most melancholy object of pity to behold; which made it even more than ever necessary that I should go to see him immediately after my arrival at Paris; in order to make such an ample and particular return as might put all former matters on a different footing, not only with him but with any other person or persons\n",
      "llama_print_timings:        load time =    1127.29 ms\n",
      "llama_print_timings:      sample time =     352.58 ms /  1024 runs   (    0.34 ms per token,  2904.31 tokens per second)\n",
      "llama_print_timings: prompt eval time =     130.76 ms /   494 tokens (    0.26 ms per token,  3777.83 tokens per second)\n",
      "llama_print_timings:        eval time =    8670.79 ms /  1023 runs   (    8.48 ms per token,   117.98 tokens per second)\n",
      "llama_print_timings:       total time =    9439.00 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "eb2e9a98-ed63-4b0b-b858-8a69c3cde67f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314384\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_0:  225 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 3.56 GiB (4.54 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   70.42 MiB\n",
      "llm_load_tensors: VRAM used           = 3577.55 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "..................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 3904.05 MiB (model: 3577.55 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under it the ancient French monarchy was soon entirely broken up; it was now no more. Neither was it likely (as in that age it might happen) that France would ever complete an income tax for herself; such a reform would be absolutely necessary to any good government of her affairs: they could not be carried on long like those of the present without some sort of income tax. The little gentry of France were all by this time completely ruined by the sumptuary laws which, during the late reign of terror, had been passed for their utter extermination. \n",
      " Scarcely less important than these papers and these bills was the following of one of those wonderful revolutions which in some countries seem to be the tendency of the age we live in; but which is perhaps unprecedented in this very respect, that there has never been a country before in which it was possible for one generation to pass away in less than half a century. \n",
      " France had now got rid of all her debts, both foreign and domestic; she had even returned the greater part of them to their respective creditors. And, although I cannot help thinking that she did not altogether succeed either in getting rid of them or returning them, she has at least this merit: She can no longer have any debt upon earth to pay. \n",
      "The author has been the principal architect of a bill introduced into both houses of Congress during the last session (which was very generally called the \"Compensated Emigration Bill\") and which, although it did not pass, still has a reasonable prospect of being brought into operation on another occasion: he therefore proposes to publish this bill in its original form; as it might possibly happen that the French Government would be inclined to give assistance towards carrying into effect any scheme which might be adopted for colonising their countrymen upon distant parts of America; and because it has been suggested, by one or two of my English friends (to whom I am greatly indebted for many most valuable hints in favour of the bill), that some such plan as is proposed in this work, might afford an excellent opportunity of rendering to France the service of sending over to her settlements upon a large scale any of their people who are disaffected or otherwise undesirable. \n",
      "The author's views were at first so much opposed by his friends and by others, that he had almost given them up as impracticable; but, being finally determined to publish the bill in its original shape, he has felt obliged to add a few pages of preliminary observations, which he trusts will prove satisfactory to the public. \n",
      "# I\n",
      "The subject of this little work may be divided into two principal parts: one is connected with the general principles by which all Governments are to be governed in modern times; and the other part with the peculiar circumstances under which any particular Government is at present placed. As both these matters are highly interesting in their separate respects, I think that some apology will be required for giving an entire dissertation upon both together: but I trust that all those persons who take a philosophical view of such questions as these, may find the subject here offered worthy of their consideration; and perhaps even the unprofessional part of my readers, when they have read to the end, may feel no regret that so much time has been employed in investigating this branch of inquiry.\n",
      "If, however, I am not allowed to publish an entire work upon such a subject as this without some apology for its intricacy and variety (both which are here necessarily produced) it will perhaps be found more proper to confine myself only to that part of the subject where any real difficulty is apprehended by persons of ordinary understanding. In doing so I shall think it necessary first, in what follows, to trace out the principles on which all civil Government has been and must be established; for these principles are generally allowed to constitute the basis upon which all modern Governments have been, or can be erected: next, as well as this, to shew how these very same principles will necessarily produce a State of things in this country which is peculiar to itself: finally, I shall conclude with offering some remarks on the advantages and disadvantages which attend different forms of civil government.\n",
      "The first part of the work may be divided into two branches; that is, first, into an examination of the principles which constitute all Governments; and secondly, into the investigation of the nature and character of that Government in particular under which this country is now placed: these, with regard to what concerns the first branch, are questions very interesting, both as they have a great tendency to throw light on many other matters, and from their bearing on those constitutional principles in general which have so lately been called forth for discussion; that is to say, whether the principles of all Government should be derived or not from the reason of mankind: And\n",
      "llama_print_timings:        load time =    1009.41 ms\n",
      "llama_print_timings:      sample time =     352.02 ms /  1024 runs   (    0.34 ms per token,  2908.93 tokens per second)\n",
      "llama_print_timings: prompt eval time =     130.57 ms /   494 tokens (    0.26 ms per token,  3783.38 tokens per second)\n",
      "llama_print_timings:        eval time =    8669.06 ms /  1023 runs   (    8.47 ms per token,   118.01 tokens per second)\n",
      "llama_print_timings:       total time =    9437.32 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c40741-4720-463b-a699-a05ae9266ab2",
   "metadata": {},
   "source": [
    "### 7B f16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e3683e44-8c9a-4f85-abfc-37c024375357",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314396\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  250.11 MiB\n",
      "llm_load_tensors: VRAM used           = 12603.02 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 12929.52 MiB (model: 12603.02 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she entertained herself, besides, with such humane achievements as sentencing a youth to have his hands cut off, his tongue torn out by the roots, and his body burned alive: all this because he had not done what the priests told him to do. She performed likewise miraculous works in the army, of which it is narrated in the history books that just at the moment when the citizens were considering quiet measures soothing to both parties, she was compelled by the fury of a warlike people whose known ruling passion is not for peace, to make a fight upon her frontiers. The quarrel had lasted already nine months, there being no end in sight to it at that moment. \n",
      " It was the very worst of times, as every schoolboy knows, when Widow Peep was left an orphan and began to work for a living; when the young man who had just failed of promotion because he had taken too little notice of his bookkeeping, got into a situation; when men of family sat down in their library in the evening after dinner without knowing how they were to get through the papers which lie on the table before them—there was no end to their troubles.\n",
      "At this period Mrs. Fezziwig kept her establishment at No. 1 Cratchit's Landing, and employed six men and Val, the boy, upon trial: also two boys—their apprentices—who were always loitering about the premises and never doing any work when there was plenty to be done; and instead of having a bell hung up for an alarm to give notice to the people who were out at a distance off when dinner was ready, she had a man in the neighborhood whose business it was to call, `Here! Here!' at the door, where the house people all broke through the wall to boil their heads and then went back to work again. \n",
      " When she came home at night her mind was much occupied with the next morning's breakfast. Her plans for the new year were carefully laid down beforehand and she had no intention of allowing any one of them to be disturbed by anyone; but there were always two or three friends staying with Mrs. Fezziwig, so that it was difficult to find quarters for them without making a mess of things: and indeed sometimes they got up in the morning looking like the people in the Goblins' Story who had been troubled in the night by hearing something begin to grow on their chairs. Mrs. Fezziwig would sit down to dinner when she felt ready, which was generally half an hour after the rest of the family; and while the meat was frying, or roasting, or broiling, according to the time of year, she would amuse herself by fitting up a new scene for the pantomime at Christmas.\n",
      "  She had a collection of old china figures which had been rescued from a baker's yard where the children's nurse had formerly worked, and they represented little scenes from well-known melodramas: she also collected some wood engravings, selected from old books, to illustrate these little plays and pantomimes. She had already introduced the pantomime in which all the characters are rats—and Mr. Rat has been made a particular favorite with her, because he was very clever at eating raisins when she was quite little—and, although it is only intended as fun for Christmas morning, one of the most popular parts of this pantomime, and indeed the whole play itself, is a song, performed by Mr. Rat himself (as Mr. Scrooge's nephew says), in which he invites the other rats to drink a glass of something strong; and which was sometimes stolen from her by an unruly boy named Peter, who would come after dark with a lighted candle to see the scenes as they were set up on table. She also had some old music books, in which she had written out several songs which Mr. Rat performed at his parties: and one of these songs was performed when Mrs. Fezziwig and I first entered into conversation at an evening party given by him a long while ago—and which I am sure you remember as well as I do.\n",
      "  But if it is true, that the more helpless we are the kinder we shall be to the helpless, the example of Mrs. Fezziwig would seem to have had an influence on her nephew's disposition. For he was a good man, Mr. Scrooge—an excellent family man; his own affairs in good order; his own domestic comforts at high levels, and his wife and daughters were happy. He took care of all the poor people who came to him for help—and when he\n",
      "llama_print_timings:        load time =   13489.39 ms\n",
      "llama_print_timings:      sample time =     357.33 ms /  1024 runs   (    0.35 ms per token,  2865.71 tokens per second)\n",
      "llama_print_timings: prompt eval time =      95.85 ms /   494 tokens (    0.19 ms per token,  5153.89 tokens per second)\n",
      "llama_print_timings:        eval time =   23140.83 ms /  1023 runs   (   22.62 ms per token,    44.21 tokens per second)\n",
      "llama_print_timings:       total time =   23880.09 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "33860261-8154-4b7c-bdec-cccc2626ba86",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314435\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  250.11 MiB\n",
      "llm_load_tensors: VRAM used           = 12603.02 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 12929.52 MiB (model: 12603.02 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors she entertained herself, while marching through Blois (the town of the Troubadours), Troyes, Dijon, Lyons, Valence, Arles, Marseilles, Nismes, Carcassonne, Narbonne, Perpignan and Figuières, - her lonely majesty quivered, like a long and slender column of cold pale blue fire, along the line we have described: reaching from the heart of the old Gaul to the frontiers of Spain, and thence branching out into Anhialt, Aragon, the valley of the Maine, Switzerland and Italy. \n",
      " During this same year of Grace one hundred and seventy-seven, France had, by a long and laborious process of slow decline, been finally reduced to a state where she was merely a nation among nations; contending successfully with none; contented to have no share in the sovereignty of Continental Europe; taking precedence of no portion of it before the restoration of her ancient independence. This state of things had given rise to symptoms which were beginning to be felt and discussed with great animation among the public men, politicians and thinkers of all parties and classes in France: such for instance as the formation of a national guard in Paris; the increasing discontent with the government expressed in the newspapers; the gatherings in large crowds at certain public places; the riots at Marseilles and Lyons; the disturbances in the southwest corner of Languedoc; and lastly the revolt against Napoleon Buonaparte which we are all more or less acquainted with. \n",
      " In the beginning of this year the French army had retired to Paris, leaving in their rear an almost desert country from Lyons to Dijon; the first battalion of the national guard were drilled under the orders of General Lafayette at Longchamp: - and it was not unreasonable to expect that before long there would be a revolution. \n",
      " It is in such an aspect of things, then, that this history begins: - with France on the eve of great events; France in the midst of her first national revival since her ancient days of glory, when she had been mistress of all Gaul and master of the Mediterranean, as far at least as Carthage and Tyre. \n",
      "Chapter First 1792 - Paris\n",
      "I HAD BEEN THREE WEEKS AT THE PATRIOT'S INN. It was about a fortnight after the King had made his solemn promise to the States-General, and four days after I came away from the Bastille with Monsieur l'Abbe, that Monsieur de Saint-Meran, my landlord, entered my room while I was at breakfast, and informed me he had orders from M. de Crosne, the minister of police, to deliver me up into his hands. The fact is, this was the day appointed for the departure of the King's family to Saint-Cloud. I was therefore preparing myself for a new change. \n",
      "I must confess that in my heart I was much more concerned about Mademoiselle Claire than about myself: for what could become of her if the King, who had promised to protect her, were removed from Paris? But Saint-Meran would not be satisfied with my answers, and gave me notice that at twelve o'clock he should expect me down stairs. He then took his leave; and I remained alone, without knowing how soon or wherefore I might be imprisoned in the Bastille a second time. \n",
      "The rest of the day passed away quietly enough, with the exception of my uneasiness about Mademoiselle Claire. When night came she did not return to her room; and at length the hours of nine struck: but no Mademoiselle Claire appeared; so I was obliged to sit up till ten before I could go to bed, where I soon fell asleep. At one o'clock in the morning, however, I awoke from a dream that disturbed me for some minutes after I waked. It appeared to my fancy I had just seen Mademoiselle Claire standing by her little desk at the farthest end of our room with tears running down her cheeks, while Monsieur le Duc de Choiseul was seated in an arm-chair before her. When they saw me awake, their countenances were both very altered; and Mademoiselle Claire, who was in tears, looked so frightened that I could not help crying out to know what had happened. \n",
      "The Duke de Choiseul then made an answer, which in the midst of my distraction seemed as if it must have been the most extraordinary thing that ever\n",
      "llama_print_timings:        load time =    4187.09 ms\n",
      "llama_print_timings:      sample time =     349.74 ms /  1024 runs   (    0.34 ms per token,  2927.92 tokens per second)\n",
      "llama_print_timings: prompt eval time =      96.38 ms /   494 tokens (    0.20 ms per token,  5125.39 tokens per second)\n",
      "llama_print_timings:        eval time =   23137.81 ms /  1023 runs   (   22.62 ms per token,    44.21 tokens per second)\n",
      "llama_print_timings:       total time =   23868.76 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "05e70274-169b-4be2-9f7c-553eb99d6361",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314465\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 291 tensors from ./models/7B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type  f16:  226 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 11008\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 7B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 6.74 B\n",
      "llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.11 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  250.11 MiB\n",
      "llm_load_tensors: VRAM used           = 12603.02 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 256.00 MB\n",
      "llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 676/676\n",
      "llama_new_context_with_model: compute buffer total size = 73.69 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 70.50 MiB\n",
      "llama_new_context_with_model: total VRAM used: 12929.52 MiB (model: 12603.02 MiB, context: 326.50 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its Christian king, the course of French carnage was more simple yet more glorious; for, while with one hand he grasped the hempen coil of his country's warlike purposes, which was tightly wound from the manufacturers up to the government arsenals, and, whence pendent by a slender thread, far away, in that distant clime where India is, he held forth with the other the alluring prize of Canada--he had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was simple; for France, while with one hand she grasped the hempen coil of her country's warlike purposes, which was tightly wound from the manufacturers up to the government arsenals and whence pendent by a slender thread, far away in that distant clime where India is--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was glorious; for France, under the guidance of her Christian King, rolling with exceeding smoothness down hill, making paper money and spending it at home--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was simple; for France, while with one hand she grasped the hempen coil of her country's warlike purposes which was tightly wound from the manufacturers up to the government arsenals and whence pendent by a slender thread far away in that distant clime where India is--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was glorious; for France, under the guidance of her Christian King, rolling with exceeding smoothness down hill making paper money and spending it at home--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was simple; for France, while with one hand she grasped the hempen coil of her country's warlike purposes which was tightly wound from the manufacturers up to the government arsenals and whence pendent by a slender thread far away in that distant clime where India is--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was glorious; for France, under the guidance of her Christian King, rolling with exceeding smoothness down hill making paper money and spending it at home--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was simple; for France, while with one hand she grasped the hempen coil of her country's warlike purposes which was tightly wound from the manufacturers up to the government arsenals and whence pendent by a slender thread far away in that distant clime where India is--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was glorious; for France, under the guidance of her Christian King, rolling with exceeding smoothness down hill making paper money and spending it at home--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was simple; for France, while with one hand she grasped the hempen coil of her country's warlike purposes which was tightly wound from the manufacturers up to the government arsenals and whence pendent by a slender thread far away in that distant clime where India is--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was glorious; for France, under the guidance of her Christian King, rolling with exceeding smoothness down hill making paper money and spending it at home--she had not so much of Asia and Africa as it was possible for human eyes to behold. Both hands were occupied. The course of French carnage was simple; for France, while with one hand she grasped the hempen coil of her country's warlike purposes which was tightly wound from the manufacturers up to the government arsenals and whence pendent by a slender thread far away in that distant clime where India is--she had not so much of Asia and Africa as it was\n",
      "llama_print_timings:        load time =    3543.21 ms\n",
      "llama_print_timings:      sample time =     348.19 ms /  1024 runs   (    0.34 ms per token,  2940.91 tokens per second)\n",
      "llama_print_timings: prompt eval time =      95.09 ms /   494 tokens (    0.19 ms per token,  5195.30 tokens per second)\n",
      "llama_print_timings:        eval time =   23151.91 ms /  1023 runs   (   22.63 ms per token,    44.19 tokens per second)\n",
      "llama_print_timings:       total time =   23883.00 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/7B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c2c1009-0642-4be0-8fee-f22a3e8bc23a",
   "metadata": {},
   "source": [
    "### 13B Q4_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "134a3c7b-30e2-460c-bfcb-ccb51c090797",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314493\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q4_0:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   88.03 MiB\n",
      "llm_load_tensors: VRAM used           = 6936.01 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 7411.01 MiB (model: 6936.01 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of its noble architect, Christopher Columbus, the great bird of American pride proceeded undaunted in its career from the Alleghanies to the Rocky Mountains; next, from the Shenandoah to the Missisippi; and still further west to the farthest recesses of the Great North-west. The exhaustless resources of the West abound in it with a magnificence to which the fabled splendour of the East has never yet attained. Spain sent out yearly to sail those seas some seven hundred ships, as the quaint old chronicles of those days inform us; while her Asiatic possessions became and remained of all others the least faltering source of her public income. The Brazil of the Western Hemisphere was already exchanging her forest leaves for woollen fabrics, and would ere long be clothed in the productions of American looms and American spindles. Let California become a second Old England, said the spirit of the old world to the spirit of the new one: let her breed a race of men, who shall give law to the future as they are now giving it to the past! And again: Let them beware how they oppress mankind on pain of retribution, let them remember that _the mills of God grind slow but sure_, that _right makes might_, that in God's court the suit must be decided some time or other.\n",
      "\n",
      "A few days after this event, a circumstance occurred which is worthy of note for its importance in relation to those by whom it was occasioned, and who are henceforward destined to take such a prominent part in our history. Amongst other things that had come under the scrutiny of General Wilkinson's eye whilst he had been stationed at Fort Massiac, was a very considerable quantity of Indian property. He found this amongst others deposited in a house, which belonged to one Lebert--a Frenchman and a resident there--who was known as an enemy to the United States, but who professed his readiness to take the oath of allegiance; which having been duly administered by Wilkinson, he delivered back the property. But here it is important that we should be able to distinguish between what actually happened, and what may have passed in Wilkinson's thoughts, for his actions do not always follow those of his mind: accordingly when he got ready for his return, there was a good deal left behind; which at that time, though the fact of his having been obliged to make restitution will be remembered by many, was no part of it. It is true that Wilkinson's character had suffered, not from this event in particular, but as it affected others of similar description, and from a suspicion that they were made with the design to have an opportunity of making use of his confidence; still there would be little or nothing of all this which he could be charged with, if Lebert's conduct had been such as to enable him to take advantage of his circumstances. But in order to do so he must have found some person in whom he could confide, and from the information of his friends he had it not long before decided on his friend and ally Mr. Wirt; who was also a resident at Fort Massiac: that gentleman had taken Wilkinson's situation much to heart; and seeing there seemed little probability of his obtaining any further advantage for himself or family by remaining, he had resolved upon going back with him into Kentucky. It will be recollected that the treaty had been negotiated on condition of the Indians delivering up their captives, and Wilkinson had sent off Lebert, to endeavour to accomplish this, at which time he received a letter from the secretary's office, informing him that one of these prisoners had been killed by his friend Wirt: but as it was not the intention of either, to be responsible for so great an act of barbarity, and as Wilkinson could not have supposed this would ever have come to his ears; he gave him up in despair, and resolved upon going with Mr. Wirt.\n",
      "\n",
      "CHAPTER XII.\n",
      "\n",
      "The next day but one after the departure of our adventurers, another party went out on a like errand; and as it will be seen in what manner they were circumstanced, they may be supposed to have been actuated by little or nothing more than mere curiosity: as Mr. Glass was about starting for New-York with a party of his friends, who wished very much to see the Falls before leaving the place; and having no better means in his power, he set out with them and their guide in the evening after their return from breakfast; having had little or nothing to do during that morning. As Mr. Wirt was now about to become a resident at Fort Mass\n",
      "llama_print_timings:        load time =    7688.70 ms\n",
      "llama_print_timings:      sample time =     355.81 ms /  1024 runs   (    0.35 ms per token,  2877.93 tokens per second)\n",
      "llama_print_timings: prompt eval time =     220.47 ms /   494 tokens (    0.45 ms per token,  2240.63 tokens per second)\n",
      "llama_print_timings:        eval time =   15022.06 ms /  1023 runs   (   14.68 ms per token,    68.10 tokens per second)\n",
      "llama_print_timings:       total time =   15883.77 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "aad047cf-d4d1-4e14-937f-1dd1e456a11b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314518\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q4_0:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   88.03 MiB\n",
      "llm_load_tensors: VRAM used           = 6936.01 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 7411.01 MiB (model: 6936.01 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she was furnished by her neighbours with bread, wine, oil, garments, and many other things, free of cost. The High Priest of the Christians wrote to the people from his golden house, that all this should be attributed to their \"piety,\" and their humble, grateful submission to the Will of Heaven, as declared by the Ambassadors in St. Pierre. In the matter of reforming abuses in other respects, the High Priest taxed the people lightly; he did not quite lay them under contribution, but rented them out at such a high price that, to use a common phrase, they would have sold their country for a song. \n",
      " All this time, there were three things which England was wishing for:—to wit, peace, plentifulness, and a reformation; these four being the cardinal points of her compass, she steered by them, in good weather or bad. So long as she had the truth of God preached to her, she felt that she could contend with any enemy; and when this was no longer the case, ministers were wanted who should reclaim the purity of doctrine from Popery: when the purity of doctrine was redeemed, it was found to be a matter very easily managed, and very inexpensive. She had always wished for peace; but so long as she saw an opportunity for doing good to herself or others at small cost, she preferred to keep her strength unimpaired, and to be prepared for any emergency that might occur. And lastly, she wished that her clergy should have better remuneration than had been usual in times past; not that they might be idle, but that being relieved from want and anxiety, they might more undistractedly apply themselves to their duties. \n",
      " On the other side of the Channel, France was suffering under an oppression greater far than any Englishman could ever have imagined. To the three objects which we have seen England wishing for, her neighbour in Europe added a fourth:—the reformation of abuses in his country. \n",
      " He also wished to be relieved from the load of taxation. He looked at England's prosperity with a jealous and envious eye; he thought that, since she enjoyed such advantages, it was monstrous that she should be so contented as not to use all her strength for their recovery: but if he had been less selfish than his neighbours were, he would have said to them, 'What does your master gain by enriching himself? You may be sure that when he has got what he wants of you, he will leave you naked and bare.' \n",
      " But the Frenchman is not so easily convinced; for if a neighbour were to take advantage of him in the matter of land or water carriage, he would call it robbery. \n",
      " At this time, however, things were changed with regard to France, and as she was suffering from an oppression greater than that which had been borne by any Englishman, so they who exercised the oppression on her did so with more impudence. \n",
      " This man is called the king; he was then a boy; but his name was Louis XV., and he reigned thirty-three years; having for father the Duke of Orleans, who had been Regent during the minority of Louis XIV., and for mother Maria Leczinska, queen of Poland, who, like many other women, wished to have her children in great places. \n",
      " As to the manner in which he was educated, I must first tell you something about his family, and afterwards will follow an account of the way in which it happened that a youth was put at the head of such an important country as France is. \n",
      " The father of the said duke was only son of the great King Henry IV. of Navarre (whom I shall have to tell you about by-and-by) and Marguerite, daughter of Henry II., king of France. \n",
      " This family of Guise being powerful at Court, he had for wife a young lady of good birth, who was sister to the Duke of Montmorency. \n",
      " The King, however, having displeased her by refusing his consent to the marriage she wanted, she left him on pretence that she did not wish to live in Paris; and as if it had been merely out of love for the King's health, she went into Artois, where her husband was governor. \n",
      " Being with child at this time, she brought forth a son (of whom we shall see an account presently) named Henry de Bourbon, who became in after years king of France by the name of Henry IV. This young nobleman had for his mother a governess called Diana of Austria. \n",
      " He was\n",
      "llama_print_timings:        load time =    2172.30 ms\n",
      "llama_print_timings:      sample time =     354.95 ms /  1024 runs   (    0.35 ms per token,  2884.94 tokens per second)\n",
      "llama_print_timings: prompt eval time =     219.26 ms /   494 tokens (    0.44 ms per token,  2253.00 tokens per second)\n",
      "llama_print_timings:        eval time =   15015.57 ms /  1023 runs   (   14.68 ms per token,    68.13 tokens per second)\n",
      "llama_print_timings:       total time =   15874.45 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "bb28eb3a-3cee-45c0-b25b-ad68ad8bdcf5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314538\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q4_0:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =   88.03 MiB\n",
      "llm_load_tensors: VRAM used           = 6936.01 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "...................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init: VRAM kv self = 400.00 MB\n",
      "llama_new_context_with_model: KV self size  =  400.00 MiB, K (f16):  200.00 MiB, V (f16):  200.00 MiB\n",
      "llama_build_graph: non-view tensors processed: 844/844\n",
      "llama_new_context_with_model: compute buffer total size = 78.19 MiB\n",
      "llama_new_context_with_model: VRAM scratch buffer: 75.00 MiB\n",
      "llama_new_context_with_model: total VRAM used: 7411.01 MiB (model: 6936.01 MiB, context: 475.00 MiB)\n",
      "\n",
      "system_info: n_threads = 32 / 64 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n",
      "sampling: \n",
      "\trepeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000\n",
      "\ttop_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 1.100\n",
      "\tmirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\n",
      "sampling order: \n",
      "CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp \n",
      "generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0\n",
      "\n",
      "\n",
      "\u001b[33m It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. \n",
      " There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. \n",
      " It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. \n",
      " France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\u001b[0m Under the guidance of her Christian pastors, she was furnished by her neighbours with bread unasked for, so well off were they at that time. Undutiful daughter! Undutiful wife! Since then, what a scene has been enacted in that once peaceful abode? The babe is turned into an old man; the sire is turned into a pensioner; and the bare bodkin-thrusts of revolution, which for centuries have made England reel like a drunken woman, have so plagued Paris as to make France stagger in her gaillardine and jerkin. \n",
      " In short, the nation had fallen into a state of mind which, if not a species of giant-madness, was a most prodigious example of human perversity; and she would, no doubt, have been soon plunged into anarchy, but for the energy of one man. And this is a truth, in reference both to the time when it occurred and when the same phenomenon has been presented under other circumstances not much unlike those which then existed: namely, in states where there has been long-standing anarchy and misgovernment. Men have called me mad; but the question is 'If I am not,' why does any body in his senses pay any regard to my madness? The question is a fair one, and I desired some friend of mine whom I trusted for his regular habits of thinking, to answer it. \n",
      " IV\n",
      " There are in London certain old people—I might well say ancient people—who are received with marked consideration, as the relics of former greatness. These persons appear as types of what has departed; and now, that the mass of society is in a state of transition, and becoming every day more and more democratic, they bear this appearance the more conspicuously. Some are so by birth, some by wealth, some by education, some by the kind of notoriety with which they have been celebrated; but all, by reason of being persons who represent something which is no longer seen. In these days it is the fashion to patronise the very poor, and not to notice the very rich. The world has got into that pitch of morality, that virtue consists in not robbing laborers, and that vice in robbing landlords. A man with forty thousand pounds a year may be supposed a very ill-used person; but as his income was not created by himself, it is plain he has no right to enjoy it. And if the gentleman in question were even to show himself the benefactor of the human race—to relieve distresses, and effect improvements, and endow colleges, or anything that is very charitable indeed—he must be deemed still more ill-used by Providence, for not having been made with a conscience. A rich man without an income is as much a pariah now, as he was the gentleman by birth in former times; when no one spoke to him, and nobody thought it worth while to tell him so. \n",
      " V\n",
      " We have all heard of the young lady who 'ran off' with a curate, having first given her father notice of her intentions in writing; which, being in form somewhat irregular, was refused; when the young woman (who was an only child), and had some five thousand pounds, carried out her own wishes. This is all very well known, but it has not prevented me from hearing this story told at least twenty times over within these last three months: each time with fresh embellishments of scandal to the father. It is very certain that a young woman's love for one of these gentle shepherds never was so much as suspected—if it had, it would have been a sufficient cause for alarming the parents. There is not perhaps in the whole round world (taking everything with sea and land into consideration) a more gloomy-minded person than an English curate; and when I say an English curate, I mean one who has lived upon his benefice and done nothing whatever during his lifetime: for I can imagine no state of life in which a clergyman is so absolutely exempt from worldly cares or responsibilities. He has no duties; he is not even supposed to know any thing about the matters of this world: all his occupation is gone if he once looks into them. And yet it will be found, I believe, that he who does most look into such matters, is generally most active and intelligent in every thing else. It is said that the clergyman at a village is not required by law to keep up any connection with his parishioners; that he may live two hundred miles off, and be little more accountable for his conduct there than if he were upon the moon: but this cannot be so in fact\n",
      "llama_print_timings:        load time =    1955.40 ms\n",
      "llama_print_timings:      sample time =     353.43 ms /  1024 runs   (    0.35 ms per token,  2897.29 tokens per second)\n",
      "llama_print_timings: prompt eval time =     217.48 ms /   494 tokens (    0.44 ms per token,  2271.43 tokens per second)\n",
      "llama_print_timings:        eval time =   15051.24 ms /  1023 runs   (   14.71 ms per token,    67.97 tokens per second)\n",
      "llama_print_timings:       total time =   15907.16 ms\n",
      "Log end\n"
     ]
    }
   ],
   "source": [
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d1552a-9361-4433-aa57-b616ecd13dee",
   "metadata": {},
   "source": [
    "### 13B f16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "24e9abc3-f9fc-4faa-9d60-d8971e686e69",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314557\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 21 key-value pairs and 363 tensors from ./models/13B-v2/ggml-model-f16.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 1\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type  f16:  282 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = F16\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 24.24 GiB (16.00 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.14 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  312.64 MiB\n",
      "llm_load_tensors: VRAM used           = 24514.08 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "..............................................................\n",
      "CUDA error 2 at ggml-cuda.cu:9081: out of memory\n",
      "current device: 0\n",
      "GGML_ASSERT: ggml-cuda.cu:9081: !\"CUDA error\"\n"
     ]
    }
   ],
   "source": [
    "# Out of memory\n",
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/13B-v2/ggml-model-f16.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04edde69-2bb1-4cd2-b0af-e265f93f128b",
   "metadata": {},
   "source": [
    "### 70B Q4_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0aac592f-7a5c-41b1-88f5-f4d3157af8b1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log start\n",
      "main: build = 1691 (7082d24)\n",
      "main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu\n",
      "main: seed  = 1703314574\n",
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 4080, compute capability 8.9\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 723 tensors from ./models/70B-v2/ggml-model-q4_0.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 8192\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 80\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 28672\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 64\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 2\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr[str,32000]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,61249]   = [\"▁ t\", \"e r\", \"i n\", \"▁ a\", \"e n...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:  161 tensors\n",
      "llama_model_loader: - type q4_0:  561 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 259/32000 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 8192\n",
      "llm_load_print_meta: n_head           = 64\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_layer          = 80\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_gqa            = 8\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 28672\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 70B\n",
      "llm_load_print_meta: model ftype      = Q4_0\n",
      "llm_load_print_meta: model params     = 68.98 B\n",
      "llm_load_print_meta: model size       = 36.20 GiB (4.51 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size       =    0.28 MiB\n",
      "llm_load_tensors: using CUDA for GPU acceleration\n",
      "llm_load_tensors: system memory used  =  140.90 MiB\n",
      "llm_load_tensors: VRAM used           = 36930.11 MiB\n",
      "llm_load_tensors: offloading 80 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 81/81 layers to GPU\n",
      ".........................................\n",
      "CUDA error 2 at ggml-cuda.cu:9081: out of memory\n",
      "current device: 0\n",
      "GGML_ASSERT: ggml-cuda.cu:9081: !\"CUDA error\"\n"
     ]
    }
   ],
   "source": [
    "# Out of memory\n",
    "!./main --color --no-mmap -ngl 10000 --temp 1.1 --repeat_penalty 1.1 -n 1024 --ignore-eos -m ./models/70B-v2/ggml-model-q4_0.gguf  -p \"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. <0x0A>\\\n",
    "There were a king with a large jaw and a queen with a plain face, on the throne of England; there were a king with a large jaw and a queen with a fair face, on the throne of France. In both countries it was clearer than crystal to the lords of the State preserves of loaves and fishes, that things in general were settled for ever. <0x0A>\\\n",
    "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual revelations were conceded to England at that favoured period, as at this. Mrs. Southcott had recently attained her five-and-twentieth blessed birthday, of whom a prophetic private in the Life Guards had heralded the sublime appearance by announcing that arrangements were made for the swallowing up of London and Westminster. Even the Cock-lane ghost had been laid only a round dozen of years, after rapping out its messages, as the spirits of this very year last past (supernaturally deficient in originality) rapped out theirs. Mere messages in the earthly order of events had lately come to the English Crown and People, from a congress of British subjects in America: which, strange to relate, have proved more important to the human race than any communications yet received through any of the chickens of the Cock-lane brood. <0x0A>\\\n",
    "France, less favoured on the whole as to matters spiritual than her sister of the shield and trident, rolled with exceeding smoothness down hill, making paper money and spending it.\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
